Sep 08 13:16:43.935465 ip-10-0-138-123 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:44.326824 ip-10-0-138-123 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:44.326824 ip-10-0-138-123 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:44.326824 ip-10-0-138-123 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:44.326824 ip-10-0-138-123 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:44.326824 ip-10-0-138-123 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:44.328384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.328298    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:44.331054 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331039    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.331054 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331055    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331061    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331065    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331069    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331072    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331075    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331078    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331081    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331084    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331087    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331090    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331093    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331095    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331098    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331100    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331103    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331112    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331115    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331117    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331120    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.331119 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331123    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331126    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331129    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331131    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331134    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331137    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331139    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331142    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331144    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331147    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331149    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331152    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331154    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331157    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331159    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331178    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331182    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331186    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331190    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331194    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.331612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331197    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331199    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331202    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331204    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331207    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331209    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331213    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331215    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331218    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331220    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331223    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331225    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331227    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331230    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331233    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331237    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331241    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331245    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331247    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331250    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.332130 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331252    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331255    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331257    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331260    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331263    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331266    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331269    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331271    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331274    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331277    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331280    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331282    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331284    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331287    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331291    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331294    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331297    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331300    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331302    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.332647 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331305    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331307    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331310    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331312    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331315    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331317    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331320    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331324    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331327    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331329    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331332    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331334    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331337    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331340    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331342    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331344    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331347    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331349    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.331352    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.333105 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332018    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332024    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332028    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332032    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332035    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332038    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332040    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332043    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332045    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332048    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332053    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332057    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332060    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332063    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332066    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332069    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332072    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332075    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.333574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332077    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332080    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332084    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332087    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332090    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332093    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332095    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332098    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332101    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332103    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332106    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332108    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332111    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332113    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332116    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332119    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332123    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332126    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332132    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.334019 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332135    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332138    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332141    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332143    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332146    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332148    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332151    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332154    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332156    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332159    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332175    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332178    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332180    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332183    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332186    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332190    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332193    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332195    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332198    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332201    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.334511 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332203    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332206    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332209    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332211    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332214    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332216    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332219    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332221    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332224    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332227    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332229    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332232    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332234    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332237    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332239    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332242    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332244    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332247    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332249    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332251    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.335009 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332254    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332257    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332259    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332261    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332264    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332266    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332269    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332272    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332275    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332277    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332280    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332283    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332285    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332287    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332290    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332293    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332297    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332299    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332301    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332304    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.335504 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332306    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332309    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332664    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332668    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332671    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332674    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332676    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332679    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332682    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332684    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332687    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332690    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332693    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332695    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332698    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332700    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332703    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332705    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332710    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.336014 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332714    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332718    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332721    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332724    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332727    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332730    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332733    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332736    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332739    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332741    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332744    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332746    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332749    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332751    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332754    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332756    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332759    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332761    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332764    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.336483 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332766    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332769    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332771    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332774    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332777    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332779    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332783    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332786    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332789    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332792    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332796    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332799    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332802    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332808    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332811    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332814    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332816    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332819    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332822    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.336949 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332824    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332827    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332830    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332832    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332835    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332838    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332840    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332843    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332845    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332848    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332850    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332853    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332855    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332859    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332862    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332864    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332867    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332869    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332871    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332874    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.337460 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332876    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332879    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332882    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332885    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332888    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332890    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332895    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332898    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332900    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332903    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332905    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332908    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332911    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332913    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332916    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332918    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332921    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332923    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332926    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332928    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.337961 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332931    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332933    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332935    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.332938    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333008    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333039    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333046    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333051    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333054    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333058    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333062    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333066    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333069    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333072    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333075    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333078    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333082    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333085    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333089    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333092    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333096    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333099    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333103    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333106    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:44.338534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333109    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333112    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333116    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333119    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333122    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333125    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333129    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333132    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333135    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333138    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333141    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333144    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333148    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333151    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333154    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333157    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333160    2580 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333178    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333183    2580 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333187    2580 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333189    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333192    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333195    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333199    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:44.339138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333202    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333205    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333208    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333211    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333214    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333217    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333220    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333222    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333225    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333228    2580 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333232    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333235    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333237    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333240    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333243    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333246    2580 flags.go:64] FLAG: --help="false"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333249    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333252    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333254    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333259    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333262    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333266    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333269    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:44.339840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333271    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333275    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333277    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333280    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333283    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333286    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333289    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333291    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333295    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333297    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333300    2580 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333303    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333306    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333309    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333314    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333317    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333320    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333323    2580 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333326    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333329    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333332    2580 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333335    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333339    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333342    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333346    2580 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:44.340518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333349    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333352    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333355    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333359    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333362    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333365    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333368    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333375    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333378    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333385    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333388    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333391    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333396    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333399    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333403    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333405    2580 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333409    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333411    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0db1dd3abf60e676f"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333415    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333418    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333421    2580 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333423    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333427    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333430    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:44.341248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333433    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333435    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333438    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333442    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333444    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333447    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333450    2580 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333453    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333456    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333459    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333461    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333465    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333468    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333471    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333474    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333477    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333480    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333483    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333486    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333489    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333492    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333495    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333500    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333503    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333506    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:44.342239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333510    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333513    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333515    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333518    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333521    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333524    2580 flags.go:64] FLAG: --v="2"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333528    2580 flags.go:64] FLAG: --version="false"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333531    2580 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333534    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.333537    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333651    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333655    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333658    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333661    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333664    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333667    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333670    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333673    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333677    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333680    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333684    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333687    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.342967 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333690    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333692    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333695    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333698    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333701    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333704    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333706    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333709    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333711    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333714    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333718    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333720    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333723    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333725    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333728    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333731    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333733    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333736    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333738    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333748    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.343562 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333751    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333754    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333756    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333759    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333762    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333764    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333767    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333769    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333774    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333777    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333779    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333782    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333784    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333787    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333789    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333792    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333795    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333797    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333800    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333803    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.344047 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333806    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333808    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333812    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333814    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333817    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333820    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333822    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333825    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333827    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333830    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333832    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333835    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333837    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333840    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333842    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333845    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333848    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333850    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333852    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333855    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.344564 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333859    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333861    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333864    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333866    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333869    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333871    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333874    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333876    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333879    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333881    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333884    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333888    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333891    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333894    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333898    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333902    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333906    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333909    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333911    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.345051 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333914    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333916    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333919    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333922    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333925    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333928    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333931    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.333933    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.334487    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.341296    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.341365    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.342149    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.342321    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342387    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342394    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.345673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342397    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342401    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342406    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342410    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342412    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342415    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342418    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342422    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342426    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342430    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342433    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342436    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342438    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342441    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342444    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342447    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342449    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342452    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342454    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.346087 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342457    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342459    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342462    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342465    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342468    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342471    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342473    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342476    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342484    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342487    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342491    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342493    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342496    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342499    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342501    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342504    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342507    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342509    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342512    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342514    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.346588 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342517    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342520    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342522    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342525    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342528    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342531    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342533    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342536    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342538    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342541    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342543    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342546    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342549    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342551    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342554    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342556    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342559    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342562    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342564    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342567    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.347153 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342571    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342574    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342578    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342580    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342583    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342586    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342588    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342591    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342593    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342596    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342599    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342601    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342604    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342606    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342609    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342611    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342614    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342617    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342620    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342623    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.347726 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342625    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342628    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342631    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342633    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342636    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342638    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342641    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342643    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342646    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342648    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342651    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342653    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342657    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342660    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342662    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342665    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342669    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.348222 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342673    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.342678    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342784    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342790    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342795    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342798    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342800    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342803    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342806    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342808    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342811    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342813    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342816    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342818    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342822    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:44.348890 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342824    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342827    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342830    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342832    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342836    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342839    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342842    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342845    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342848    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342850    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342853    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342855    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342858    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342861    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342863    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342865    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342868    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342871    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342873    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342876    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:44.349522 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342878    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342881    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342884    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342886    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342889    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342892    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342894    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342897    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342899    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342901    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342904    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342907    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342910    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342912    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342915    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342917    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342922    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342924    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342927    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:44.350068 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342929    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342932    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342935    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342937    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342939    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342942    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342945    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342947    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342950    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342953    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342955    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342958    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342960    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342962    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342965    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342968    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342970    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342973    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342976    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:44.350684 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342979    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342982    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342984    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342987    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342989    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342992    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342995    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.342997    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343000    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343002    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343005    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343007    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343009    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343012    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343015    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343017    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343019    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343022    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343025    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343027    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:44.351215 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343030    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343032    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343035    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343037    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343041    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343045    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:44.343048    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.343053    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.343178    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.345637    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.346589    2580 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.346700    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.347422    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:44.351728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.349215    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:44.352061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.350380    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:44.359953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.359936    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:44.362532 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.362518    2580 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:44.363373 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.363353    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:44.367020 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.366999    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ae6315e1-9049-46c6-bcd3-48314111f6db:/dev/nvme0n1p3 b14f2840-098c-4ee1-ac29-0cc24f8c59d5:/dev/nvme0n1p4]
Sep 08 13:16:44.367094 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.367019    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:44.373254 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.373119    2580 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:44.371862393 +0000 UTC m=+0.322516336 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116468 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2638402ea9522a938ddf84e802e24d SystemUUID:ec263840-2ea9-522a-938d-df84e802e24d BootID:f4ea81c3-0058-43e0-a4a4-ebc955e79192 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:a5:e4:20:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:a5:e4:20:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:7d:31:95:e2:a8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:44.373254 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.373244    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:44.373416 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.373383    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:44.376717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.376695    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:44.376889 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.376720    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:44.377394 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.377383    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:44.377458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.377399    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:44.377458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.377412    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:44.378122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.378110    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:44.378870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.378859    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:44.379677 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.379660    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:44.381861 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.381848    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:44.381991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.381980    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:44.382047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.382021    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:44.382047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.382039    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:44.382135 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.382055    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:44.382135 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.382068    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:44.382135 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.382080    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:44.384874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.384856    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:44.384998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.384988    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:44.387628 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.387613    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:44.389721 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.389707    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:44.389777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.389737    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:44.390935 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390921    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:44.390985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390947    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:44.390985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390964    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:44.390985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390972    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:44.390985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390981    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.390986    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391000    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391005    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391011    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391017    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391025    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:44.391116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391031    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:44.391752 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391737    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:44.391798 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.391758    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:44.391842 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.391821    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:44.391920 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.391900    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:44.395214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.395198    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:44.395331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.395320    2580 server.go:1268] "Started kubelet"
Sep 08 13:16:44.395442 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.395401    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:44.395505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.395411    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:44.395505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.395471    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:44.396197 ip-10-0-138-123 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:44.399131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.399094    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:44.399321 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.399303    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:44.400489 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.400469    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:44.402434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.402416    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:44.402523 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.402449    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:44.402672 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.402648    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.403213 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403201    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:44.403516 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403501    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:44.403597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403466    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:44.403597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403576    2580 factory.go:55] Registering systemd factory
Sep 08 13:16:44.403597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403586    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:44.403747 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403609    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:44.403747 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403617    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:44.403747 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403317    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:44.403875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403847    2580 factory.go:153] Registering CRI-O factory
Sep 08 13:16:44.403875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403858    2580 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:44.403875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403874    2580 factory.go:103] Registering Raw factory
Sep 08 13:16:44.403982 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.403884    2580 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:44.404491 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.404476    2580 manager.go:349] Starting recovery of all containers
Sep 08 13:16:44.406666 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.405658    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-123.ec2.internal.18d35ae229201327  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-123.ec2.internal,UID:ip-10-0-138-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-123.ec2.internal,},FirstTimestamp:2026-09-08 13:16:44.395295527 +0000 UTC m=+0.345949470,LastTimestamp:2026-09-08 13:16:44.395295527 +0000 UTC m=+0.345949470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-123.ec2.internal,}"
Sep 08 13:16:44.406792 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.406775    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:44.406887 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.406854    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:44.407854 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.407373    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wgrqf"
Sep 08 13:16:44.414350 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.414332    2580 manager.go:354] Recovery completed
Sep 08 13:16:44.415686 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.415665    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wgrqf"
Sep 08 13:16:44.418705 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.418690    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.421137 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421120    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.421224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421148    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.421224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421159    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.421708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421694    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:44.421708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421705    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.421708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421709    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:44.421824 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.421727    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:44.422785 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.422770    2580 policy_none.go:49] "None policy: Start"
Sep 08 13:16:44.422785 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.422786    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:44.422896 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.422795    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:44.423305 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.423240    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-123.ec2.internal.18d35ae22aaa5a93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-123.ec2.internal,UID:ip-10-0-138-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-123.ec2.internal,},FirstTimestamp:2026-09-08 13:16:44.421134995 +0000 UTC m=+0.371788938,LastTimestamp:2026-09-08 13:16:44.421134995 +0000 UTC m=+0.371788938,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-123.ec2.internal,}"
Sep 08 13:16:44.423724 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.423713    2580 policy_none.go:47] "Start"
Sep 08 13:16:44.428856 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.428830    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.428936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.428867    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.428936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.428885    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.450870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.450746    2580 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:44.450994 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.450897    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:44.450994 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.450910    2580 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:44.451152 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.451139    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:44.451236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.451154    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:44.451319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.451306    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:44.451400 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.451380    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:44.451400 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.451391    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:44.451805 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.451757    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:44.451896 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.451811    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.528411 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.528385    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:44.529539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.529525    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:44.529612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.529569    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:44.529612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.529599    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:44.529703 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.529687    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:44.531790 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.531771    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:44.551726 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.551709    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.552691 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.552672    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.552774 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.552707    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.552774 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.552722    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.552774 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.552743    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.559143 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.559127    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.559228 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.559147    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-123.ec2.internal\": node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.581939 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.581872    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.630116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.630092    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"]
Sep 08 13:16:44.630210 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.630156    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.631538 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.631523    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.631624 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.631554    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.631624 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.631568    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.633712 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.633698    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.633836 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.633818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.633904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.633855    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.634400 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634383    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.634476 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634409    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.634476 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634424    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.634573 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634492    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.634573 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634524    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.634573 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.634538    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.636656 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.636641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.636736 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.636665    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:44.637340 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.637324    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:44.637435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.637356    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:44.637435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.637369    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:44.651771 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.651749    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.655106 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.655091    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.681972 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.681955    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.705084 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.705064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.705153 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.705087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7691c42ef3bfce427afbddd33d4f4b3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"b7691c42ef3bfce427afbddd33d4f4b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.705153 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.705105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.782979 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.782945    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.805375 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.805452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.805452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.805452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7691c42ef3bfce427afbddd33d4f4b3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"b7691c42ef3bfce427afbddd33d4f4b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.805584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7691c42ef3bfce427afbddd33d4f4b3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"b7691c42ef3bfce427afbddd33d4f4b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.805584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.805523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd25bbcec769cf1adcfbe899ed035566-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"cd25bbcec769cf1adcfbe899ed035566\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.883634 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.883556    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:44.953253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.953217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.958094 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:44.958068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:44.984273 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:44.983668    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.084244 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.084205    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.189161 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.189095    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.289952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.289900    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.347461 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.347436    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:45.348074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.347572    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:45.390131 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.390102    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.419080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.419044    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:44 +0000 UTC" deadline="2028-06-03 12:02:20.166694382 +0000 UTC"
Sep 08 13:16:45.419080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.419070    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15214h45m34.747627445s"
Sep 08 13:16:45.491449 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.491415    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.571287 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.571256    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:45.593055 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:45.593028    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Sep 08 13:16:45.611316 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.611288    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:45.703707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.703678    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:45.717224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.717193    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:45.719259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.719237    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Sep 08 13:16:45.720498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.720481    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:45.732859 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:45.732841    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:46.371051 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.371018    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:46.384732 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.384709    2580 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:46.394702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.394678    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:46.396502 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.396478    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7sl66","openshift-image-registry/node-ca-2d8d5","openshift-multus/multus-x52hp","openshift-network-operator/iptables-alerter-fqt4r","openshift-ovn-kubernetes/ovnkube-node-lv5hb","kube-system/global-pull-secret-syncer-fv5ww","kube-system/konnectivity-agent-mkmcw","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5p97k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-mfl22","openshift-multus/network-metrics-daemon-4q54b","openshift-network-diagnostics/network-check-target-8cbrp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"]
Sep 08 13:16:46.398991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.398969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.401250 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.401218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:46.401332 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.401293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:46.402829 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.402809    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:46.402928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.402832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:46.402928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.402812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4t8t\""
Sep 08 13:16:46.403407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.403389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.403548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.403528    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:46.405750 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.405730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.407439 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.407418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:46.407811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.407785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:46.407871 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.407830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cttmh\""
Sep 08 13:16:46.407957 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.407940    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.408028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.407948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.408272 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.408254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.409122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.409082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7v6xz\""
Sep 08 13:16:46.409122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.409104    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.409327 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.409120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.409327 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.409140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:46.410666 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.410648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.410761 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.410726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:46.410830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.410775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.411264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.411234    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:46.411377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.411356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:46.411438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.411374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.412978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-hostroot\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-etc-kubernetes\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fcd4a12-9080-4095-9e1f-28c1107d6de5-host-slash\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckl96\" (UniqueName: \"kubernetes.io/projected/5fcd4a12-9080-4095-9e1f-28c1107d6de5-kube-api-access-ckl96\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3af11ec-f900-46e3-a79d-c54f27ef55f4-agent-certs\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3af11ec-f900-46e3-a79d-c54f27ef55f4-konnectivity-ca\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cnibin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cni-binary-copy\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-k8s-cni-cncf-io\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-netns\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-multus\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-system-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-conf-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-daemon-config\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.414430 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-multus-certs\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvlt\" (UniqueName: \"kubernetes.io/projected/684f95d3-ca3e-45fe-8dad-9487ee06e49d-kube-api-access-2dvlt\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fcd4a12-9080-4095-9e1f-28c1107d6de5-iptables-alerter-script\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-bin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-socket-dir-parent\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.413958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-kubelet\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.414532    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.414568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-42wsg\""
Sep 08 13:16:46.415147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.414651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:46.415627 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.415597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.416747 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.416705    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:46.417928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.417906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gqfzt\""
Sep 08 13:16:46.418030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.417993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.418109 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.418027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.418207 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.418189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.420325 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.420309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.420640 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.420618    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-892xm\""
Sep 08 13:16:46.420750 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.420735    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.420869 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.420854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.422651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.422631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.422842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.422822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:46.422968 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.422951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.423296 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.423280    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.423395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.423279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qvngp\""
Sep 08 13:16:46.425205 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.425169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.425329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.425264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:46.425614 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.425593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v5j7v\""
Sep 08 13:16:46.425702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.425657    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.425749 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.425733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.427718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.427700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.427818 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.427763    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:46.428064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.428034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:46.428144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.428122    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqfzw\""
Sep 08 13:16:46.428749 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.428731    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:46.439466 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.439446    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjjbn"
Sep 08 13:16:46.447819 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.447801    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjjbn"
Sep 08 13:16:46.506514 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.506491    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:46.514906 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.514883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.514995 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.514918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-systemd-units\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.514995 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.514937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-node-log\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.514995 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.514963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-env-overrides\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.515164 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3af11ec-f900-46e3-a79d-c54f27ef55f4-konnectivity-ca\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.515164 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-k8s-cni-cncf-io\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515164 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-conf-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515164 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-cnibin\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.515164 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysconfig\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-k8s-cni-cncf-io\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-conf-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-multus-certs\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-netd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-multus-certs\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-bin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-bin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc1b9c76-e22c-49f3-858f-99ffaa379061-host\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-log-socket\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-socket-dir-parent\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-kubelet\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-etc-kubernetes\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.515438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-socket-dir-parent\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-socket-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-kubelet\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-etc-kubernetes\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-systemd\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f3af11ec-f900-46e3-a79d-c54f27ef55f4-konnectivity-ca\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-host\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/272ac4cc-2f4d-45de-b56b-e83b1918116c-hosts-file\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckl96\" (UniqueName: \"kubernetes.io/projected/5fcd4a12-9080-4095-9e1f-28c1107d6de5-kube-api-access-ckl96\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc1b9c76-e22c-49f3-858f-99ffaa379061-serviceca\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-netns\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-multus\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl5b\" (UniqueName: \"kubernetes.io/projected/dcabf8af-b206-4d16-b396-8a875b739614-kube-api-access-fgl5b\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-var-lib-cni-multus\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-conf\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-tuned\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-host-run-netns\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-netns\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-system-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xf4\" (UniqueName: \"kubernetes.io/projected/cc1b9c76-e22c-49f3-858f-99ffaa379061-kube-api-access-z8xf4\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfbh\" (UniqueName: \"kubernetes.io/projected/228911e2-1e89-4968-9b8f-bb2d6f597d2e-kube-api-access-9wfbh\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-system-cni-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-ovn\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-system-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.515981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-lib-modules\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb50200c-d163-4934-9370-389b297bf6b3-ovn-node-metrics-cert\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-script-lib\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3af11ec-f900-46e3-a79d-c54f27ef55f4-agent-certs\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cnibin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cni-binary-copy\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.516866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-daemon-config\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvlt\" (UniqueName: \"kubernetes.io/projected/684f95d3-ca3e-45fe-8dad-9487ee06e49d-kube-api-access-2dvlt\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cnibin\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-registration-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-systemd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-bin\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9l6c\" (UniqueName: \"kubernetes.io/projected/58afe534-f025-4185-a57b-208bd1c6312a-kube-api-access-f9l6c\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-kubernetes\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-var-lib-kubelet\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-var-lib-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272ac4cc-2f4d-45de-b56b-e83b1918116c-tmp-dir\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-kubelet\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-slash\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.517615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlgj\" (UniqueName: \"kubernetes.io/projected/bb50200c-d163-4934-9370-389b297bf6b3-kube-api-access-mdlgj\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-kubelet-config\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fcd4a12-9080-4095-9e1f-28c1107d6de5-host-slash\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-daemon-config\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516721    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fcd4a12-9080-4095-9e1f-28c1107d6de5-host-slash\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fcd4a12-9080-4095-9e1f-28c1107d6de5-iptables-alerter-script\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-multus-cni-dir\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-run\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-dbus\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/684f95d3-ca3e-45fe-8dad-9487ee06e49d-cni-binary-copy\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-sys\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.516991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgxh\" (UniqueName: \"kubernetes.io/projected/05fa069d-7b89-4ce6-823f-613bfb31539a-kube-api-access-fqgxh\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.518298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-etc-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-sys-fs\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-modprobe-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fcd4a12-9080-4095-9e1f-28c1107d6de5-iptables-alerter-script\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-tmp\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8mt\" (UniqueName: \"kubernetes.io/projected/272ac4cc-2f4d-45de-b56b-e83b1918116c-kube-api-access-gr8mt\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-config\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-hostroot\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-device-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.518958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.517409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/684f95d3-ca3e-45fe-8dad-9487ee06e49d-hostroot\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.520591 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.520573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f3af11ec-f900-46e3-a79d-c54f27ef55f4-agent-certs\") pod \"konnectivity-agent-mkmcw\" (UID: \"f3af11ec-f900-46e3-a79d-c54f27ef55f4\") " pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.534410 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.534387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckl96\" (UniqueName: \"kubernetes.io/projected/5fcd4a12-9080-4095-9e1f-28c1107d6de5-kube-api-access-ckl96\") pod \"iptables-alerter-fqt4r\" (UID: \"5fcd4a12-9080-4095-9e1f-28c1107d6de5\") " pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.534536 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.534518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvlt\" (UniqueName: \"kubernetes.io/projected/684f95d3-ca3e-45fe-8dad-9487ee06e49d-kube-api-access-2dvlt\") pod \"multus-x52hp\" (UID: \"684f95d3-ca3e-45fe-8dad-9487ee06e49d\") " pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.534703 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.534689    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:46.534748 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.534707    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:46.534748 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.534730    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:46.534859 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.534788    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.034772743 +0000 UTC m=+2.985426678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:46.553999 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.553959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7691c42ef3bfce427afbddd33d4f4b3.slice/crio-3dbc9b81f954d7a72c8c6154a16a37aa0a0a73da68f6f48093a83462ea1a4ce5 WatchSource:0}: Error finding container 3dbc9b81f954d7a72c8c6154a16a37aa0a0a73da68f6f48093a83462ea1a4ce5: Status 404 returned error can't find the container with id 3dbc9b81f954d7a72c8c6154a16a37aa0a0a73da68f6f48093a83462ea1a4ce5
Sep 08 13:16:46.558334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.558318    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:46.617641 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc1b9c76-e22c-49f3-858f-99ffaa379061-host\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-log-socket\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-socket-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-systemd\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-host\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.617757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc1b9c76-e22c-49f3-858f-99ffaa379061-host\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-log-socket\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-systemd\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-host\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/272ac4cc-2f4d-45de-b56b-e83b1918116c-hosts-file\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc1b9c76-e22c-49f3-858f-99ffaa379061-serviceca\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl5b\" (UniqueName: \"kubernetes.io/projected/dcabf8af-b206-4d16-b396-8a875b739614-kube-api-access-fgl5b\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/272ac4cc-2f4d-45de-b56b-e83b1918116c-hosts-file\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-conf\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-socket-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.617970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-tuned\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-netns\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-netns\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xf4\" (UniqueName: \"kubernetes.io/projected/cc1b9c76-e22c-49f3-858f-99ffaa379061-kube-api-access-z8xf4\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-conf\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfbh\" (UniqueName: \"kubernetes.io/projected/228911e2-1e89-4968-9b8f-bb2d6f597d2e-kube-api-access-9wfbh\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-system-cni-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc1b9c76-e22c-49f3-858f-99ffaa379061-serviceca\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-ovn\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-system-cni-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-lib-modules\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-ovn\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb50200c-d163-4934-9370-389b297bf6b3-ovn-node-metrics-cert\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-script-lib\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.618701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-lib-modules\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-registration-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-systemd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-bin\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9l6c\" (UniqueName: \"kubernetes.io/projected/58afe534-f025-4185-a57b-208bd1c6312a-kube-api-access-f9l6c\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-registration-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-kubernetes\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-systemd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-kubernetes\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-run-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-bin\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.619690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-var-lib-kubelet\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.618983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-var-lib-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272ac4cc-2f4d-45de-b56b-e83b1918116c-tmp-dir\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-var-lib-kubelet\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-kubelet\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-slash\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-var-lib-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlgj\" (UniqueName: \"kubernetes.io/projected/bb50200c-d163-4934-9370-389b297bf6b3-kube-api-access-mdlgj\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-kubelet-config\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-kubelet\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-slash\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-kubelet-config\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-run\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-script-lib\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-run\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-dbus\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.620593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-sys\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgxh\" (UniqueName: \"kubernetes.io/projected/05fa069d-7b89-4ce6-823f-613bfb31539a-kube-api-access-fqgxh\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.619396    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-sys\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cd3528-9abe-4b26-8744-c662668b3f60-dbus\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272ac4cc-2f4d-45de-b56b-e83b1918116c-tmp-dir\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-etc-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.619459    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.119439975 +0000 UTC m=+3.070093935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-etc-openvswitch\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysctl-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.619487    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-sys-fs\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-modprobe-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-tmp\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-sys-fs\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.621671 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8mt\" (UniqueName: \"kubernetes.io/projected/272ac4cc-2f4d-45de-b56b-e83b1918116c-kube-api-access-gr8mt\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:46.619679    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.119659091 +0000 UTC m=+3.070313036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-config\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-modprobe-d\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-device-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-systemd-units\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-node-log\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-env-overrides\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-cnibin\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcabf8af-b206-4d16-b396-8a875b739614-cnibin\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-node-log\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.619976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58afe534-f025-4185-a57b-208bd1c6312a-device-dir\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-systemd-units\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysconfig\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-sysconfig\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-netd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb50200c-d163-4934-9370-389b297bf6b3-host-cni-netd\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-ovnkube-config\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb50200c-d163-4934-9370-389b297bf6b3-env-overrides\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-etc-tuned\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.620965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcabf8af-b206-4d16-b396-8a875b739614-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.621852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/228911e2-1e89-4968-9b8f-bb2d6f597d2e-tmp\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.622904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.622371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb50200c-d163-4934-9370-389b297bf6b3-ovn-node-metrics-cert\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.628027 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.628002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl5b\" (UniqueName: \"kubernetes.io/projected/dcabf8af-b206-4d16-b396-8a875b739614-kube-api-access-fgl5b\") pod \"multus-additional-cni-plugins-mfl22\" (UID: \"dcabf8af-b206-4d16-b396-8a875b739614\") " pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.628136 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.628032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9l6c\" (UniqueName: \"kubernetes.io/projected/58afe534-f025-4185-a57b-208bd1c6312a-kube-api-access-f9l6c\") pod \"aws-ebs-csi-driver-node-hg5z9\" (UID: \"58afe534-f025-4185-a57b-208bd1c6312a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.628334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.628316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xf4\" (UniqueName: \"kubernetes.io/projected/cc1b9c76-e22c-49f3-858f-99ffaa379061-kube-api-access-z8xf4\") pod \"node-ca-2d8d5\" (UID: \"cc1b9c76-e22c-49f3-858f-99ffaa379061\") " pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.629492 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.629456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfbh\" (UniqueName: \"kubernetes.io/projected/228911e2-1e89-4968-9b8f-bb2d6f597d2e-kube-api-access-9wfbh\") pod \"tuned-5p97k\" (UID: \"228911e2-1e89-4968-9b8f-bb2d6f597d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.629707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.629685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgxh\" (UniqueName: \"kubernetes.io/projected/05fa069d-7b89-4ce6-823f-613bfb31539a-kube-api-access-fqgxh\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:46.630166 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.630149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlgj\" (UniqueName: \"kubernetes.io/projected/bb50200c-d163-4934-9370-389b297bf6b3-kube-api-access-mdlgj\") pod \"ovnkube-node-lv5hb\" (UID: \"bb50200c-d163-4934-9370-389b297bf6b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.630753 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.630733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8mt\" (UniqueName: \"kubernetes.io/projected/272ac4cc-2f4d-45de-b56b-e83b1918116c-kube-api-access-gr8mt\") pod \"node-resolver-7sl66\" (UID: \"272ac4cc-2f4d-45de-b56b-e83b1918116c\") " pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.727405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.727375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:16:46.738541 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.738512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3af11ec_f900_46e3_a79d_c54f27ef55f4.slice/crio-dc3c33bfa80b9ddb477a66b113db8a809ad8e8e3b7034ed7c0f3eea34f41f6b2 WatchSource:0}: Error finding container dc3c33bfa80b9ddb477a66b113db8a809ad8e8e3b7034ed7c0f3eea34f41f6b2: Status 404 returned error can't find the container with id dc3c33bfa80b9ddb477a66b113db8a809ad8e8e3b7034ed7c0f3eea34f41f6b2
Sep 08 13:16:46.745253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.745220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x52hp"
Sep 08 13:16:46.754291 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.754261    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684f95d3_ca3e_45fe_8dad_9487ee06e49d.slice/crio-3cffe83bf8890fe76d2158dd4420d935cd0c556c50380b3eb4e8cf914ad27e6e WatchSource:0}: Error finding container 3cffe83bf8890fe76d2158dd4420d935cd0c556c50380b3eb4e8cf914ad27e6e: Status 404 returned error can't find the container with id 3cffe83bf8890fe76d2158dd4420d935cd0c556c50380b3eb4e8cf914ad27e6e
Sep 08 13:16:46.759545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.759525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqt4r"
Sep 08 13:16:46.764493 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.764475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:16:46.767412 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.767391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcd4a12_9080_4095_9e1f_28c1107d6de5.slice/crio-9431dc5278dacd0d5364d3eaa1d0453d60ae52a5f75feb6fbc0239edfab7e337 WatchSource:0}: Error finding container 9431dc5278dacd0d5364d3eaa1d0453d60ae52a5f75feb6fbc0239edfab7e337: Status 404 returned error can't find the container with id 9431dc5278dacd0d5364d3eaa1d0453d60ae52a5f75feb6fbc0239edfab7e337
Sep 08 13:16:46.772982 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.772959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb50200c_d163_4934_9370_389b297bf6b3.slice/crio-e37d720062eac211ed9d716ed5befd484013033106b288e1c2000a1d9eeb0524 WatchSource:0}: Error finding container e37d720062eac211ed9d716ed5befd484013033106b288e1c2000a1d9eeb0524: Status 404 returned error can't find the container with id e37d720062eac211ed9d716ed5befd484013033106b288e1c2000a1d9eeb0524
Sep 08 13:16:46.778943 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.778925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p97k"
Sep 08 13:16:46.784587 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.784571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7sl66"
Sep 08 13:16:46.788822 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.788735    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228911e2_1e89_4968_9b8f_bb2d6f597d2e.slice/crio-331f983fa49a7b9ce28c40ad2cb097b614e9a78f2ad38b206d808e984712724a WatchSource:0}: Error finding container 331f983fa49a7b9ce28c40ad2cb097b614e9a78f2ad38b206d808e984712724a: Status 404 returned error can't find the container with id 331f983fa49a7b9ce28c40ad2cb097b614e9a78f2ad38b206d808e984712724a
Sep 08 13:16:46.794673 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.794654    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272ac4cc_2f4d_45de_b56b_e83b1918116c.slice/crio-a811b26d3aa526d0114612147fe0abed7dd6310983a60a3bde2aa25d06b71029 WatchSource:0}: Error finding container a811b26d3aa526d0114612147fe0abed7dd6310983a60a3bde2aa25d06b71029: Status 404 returned error can't find the container with id a811b26d3aa526d0114612147fe0abed7dd6310983a60a3bde2aa25d06b71029
Sep 08 13:16:46.799043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.799025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2d8d5"
Sep 08 13:16:46.807929 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.807908    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1b9c76_e22c_49f3_858f_99ffaa379061.slice/crio-ae6e3cbe748d77e72ebc3726fef841bf24f926ec8969f9b3d84fc2c6fdb94eb8 WatchSource:0}: Error finding container ae6e3cbe748d77e72ebc3726fef841bf24f926ec8969f9b3d84fc2c6fdb94eb8: Status 404 returned error can't find the container with id ae6e3cbe748d77e72ebc3726fef841bf24f926ec8969f9b3d84fc2c6fdb94eb8
Sep 08 13:16:46.815123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.815102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9"
Sep 08 13:16:46.821536 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:46.821519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfl22"
Sep 08 13:16:46.823250 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.823228    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58afe534_f025_4185_a57b_208bd1c6312a.slice/crio-d4d95b27a2f050a6e4a28b76bfb5b22bc6f8c14564a59edee0876ebebb4a2256 WatchSource:0}: Error finding container d4d95b27a2f050a6e4a28b76bfb5b22bc6f8c14564a59edee0876ebebb4a2256: Status 404 returned error can't find the container with id d4d95b27a2f050a6e4a28b76bfb5b22bc6f8c14564a59edee0876ebebb4a2256
Sep 08 13:16:46.829872 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:46.829852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcabf8af_b206_4d16_b396_8a875b739614.slice/crio-b11e205c7e19d3a3b05aaf091f8b40b481595298f0b35b4fba41716b88d9c91c WatchSource:0}: Error finding container b11e205c7e19d3a3b05aaf091f8b40b481595298f0b35b4fba41716b88d9c91c: Status 404 returned error can't find the container with id b11e205c7e19d3a3b05aaf091f8b40b481595298f0b35b4fba41716b88d9c91c
Sep 08 13:16:47.124222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.124162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:47.124222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.124225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.124270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124341    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124390    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124410    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.12439917 +0000 UTC m=+4.075053125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124428    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124444    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124450    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.124432792 +0000 UTC m=+4.075086734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:47.124488 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:47.124483    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.124472778 +0000 UTC m=+4.075126710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:47.290204 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:16:47.290152    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd25bbcec769cf1adcfbe899ed035566.slice/crio-16cfee0f1be4b0c3c0dd94ab3da4ae2d72708baeff547c788ad113ba0364f1ea WatchSource:0}: Error finding container 16cfee0f1be4b0c3c0dd94ab3da4ae2d72708baeff547c788ad113ba0364f1ea: Status 404 returned error can't find the container with id 16cfee0f1be4b0c3c0dd94ab3da4ae2d72708baeff547c788ad113ba0364f1ea
Sep 08 13:16:47.452006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.451825    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:46 +0000 UTC" deadline="2028-03-09 05:38:06.166562572 +0000 UTC"
Sep 08 13:16:47.452006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.451896    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13144h21m18.714671187s"
Sep 08 13:16:47.538584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.538427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerStarted","Data":"b11e205c7e19d3a3b05aaf091f8b40b481595298f0b35b4fba41716b88d9c91c"}
Sep 08 13:16:47.545479 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.545446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9" event={"ID":"58afe534-f025-4185-a57b-208bd1c6312a","Type":"ContainerStarted","Data":"d4d95b27a2f050a6e4a28b76bfb5b22bc6f8c14564a59edee0876ebebb4a2256"}
Sep 08 13:16:47.550300 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.550269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2d8d5" event={"ID":"cc1b9c76-e22c-49f3-858f-99ffaa379061","Type":"ContainerStarted","Data":"ae6e3cbe748d77e72ebc3726fef841bf24f926ec8969f9b3d84fc2c6fdb94eb8"}
Sep 08 13:16:47.557575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.557546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p97k" event={"ID":"228911e2-1e89-4968-9b8f-bb2d6f597d2e","Type":"ContainerStarted","Data":"331f983fa49a7b9ce28c40ad2cb097b614e9a78f2ad38b206d808e984712724a"}
Sep 08 13:16:47.573344 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.573312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"e37d720062eac211ed9d716ed5befd484013033106b288e1c2000a1d9eeb0524"}
Sep 08 13:16:47.577535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.577481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqt4r" event={"ID":"5fcd4a12-9080-4095-9e1f-28c1107d6de5","Type":"ContainerStarted","Data":"9431dc5278dacd0d5364d3eaa1d0453d60ae52a5f75feb6fbc0239edfab7e337"}
Sep 08 13:16:47.580156 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.580111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"b7691c42ef3bfce427afbddd33d4f4b3","Type":"ContainerStarted","Data":"3dbc9b81f954d7a72c8c6154a16a37aa0a0a73da68f6f48093a83462ea1a4ce5"}
Sep 08 13:16:47.585702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.585675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7sl66" event={"ID":"272ac4cc-2f4d-45de-b56b-e83b1918116c","Type":"ContainerStarted","Data":"a811b26d3aa526d0114612147fe0abed7dd6310983a60a3bde2aa25d06b71029"}
Sep 08 13:16:47.597683 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.597659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x52hp" event={"ID":"684f95d3-ca3e-45fe-8dad-9487ee06e49d","Type":"ContainerStarted","Data":"3cffe83bf8890fe76d2158dd4420d935cd0c556c50380b3eb4e8cf914ad27e6e"}
Sep 08 13:16:47.599362 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.599336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkmcw" event={"ID":"f3af11ec-f900-46e3-a79d-c54f27ef55f4","Type":"ContainerStarted","Data":"dc3c33bfa80b9ddb477a66b113db8a809ad8e8e3b7034ed7c0f3eea34f41f6b2"}
Sep 08 13:16:47.621206 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:47.621160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"cd25bbcec769cf1adcfbe899ed035566","Type":"ContainerStarted","Data":"16cfee0f1be4b0c3c0dd94ab3da4ae2d72708baeff547c788ad113ba0364f1ea"}
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.134310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.134365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.134422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.134577    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.134596    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.134617    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.134709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.134662663 +0000 UTC m=+6.085316614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.135142    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.135198    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.135181325 +0000 UTC m=+6.085835258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.135279    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:48.135375 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.135318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.135304572 +0000 UTC m=+6.085958507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:48.452520 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.452426    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:46 +0000 UTC" deadline="2028-04-01 11:40:09.333488855 +0000 UTC"
Sep 08 13:16:48.452520 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.452469    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13702h23m20.881024219s"
Sep 08 13:16:48.532658 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.532624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:48.532833 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.532758    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:48.533194 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.533166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:48.533320 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.533286    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:48.533382 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:48.533373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:48.533481 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:48.533462    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:50.155126 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.155046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:50.155615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.155199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:50.155615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.155239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:50.156421 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156391    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:50.156593 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156432    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:50.156593 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156460    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:50.156593 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156496    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:50.156593 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156539    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.156510085 +0000 UTC m=+10.107164044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:50.156593 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.156571    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.156552302 +0000 UTC m=+10.107206252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:50.157160 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.157137    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:50.157280 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.157226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.157209453 +0000 UTC m=+10.107863404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:50.530864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.530331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:50.530864 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.530477    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:50.530864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.530580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:50.530864 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.530708    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:50.530864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:50.530784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:50.531163 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:50.530881    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:52.533801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:52.533773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:52.534156 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:52.533907    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:52.534156 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:52.534022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:52.534156 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:52.534102    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:52.534329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:52.534317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:52.534443 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:52.534421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:54.192436 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.192131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.192499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.192531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192308    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192584    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192602    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192638    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192643    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192660    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.192640291 +0000 UTC m=+18.143294223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.192669547 +0000 UTC m=+18.143323476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:54.192913 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.192693    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.192684095 +0000 UTC m=+18.143338026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:54.530344 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.530308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:54.530519 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.530439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:54.530808 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.530787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:54.530912 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.530893    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:54.530982 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:54.530801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:54.531038 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:54.531006    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:56.532938 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:56.532860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:56.532938 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:56.532882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:56.533413 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:56.532860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:56.533413 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:56.532978    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:56.533413 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:56.533058    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:16:56.533413 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:56.533118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:58.530161 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:58.530122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:16:58.530613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:58.530122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:16:58.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:58.530256    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:16:58.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:58.530329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:16:58.530613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:16:58.530122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:16:58.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:16:58.530419    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:00.530782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:00.530659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:00.530782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:00.530671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:00.531327 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:00.530799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:00.531327 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:00.530801    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:00.531327 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:00.530952    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:00.531327 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:00.531030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:02.253463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.253424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.253486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.253543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253608    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253667    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253687    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.253668373 +0000 UTC m=+34.204322323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253690    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253713    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253749    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.253739116 +0000 UTC m=+34.204393059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253673    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:02.253952 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.253783    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.253774563 +0000 UTC m=+34.204428494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:02.530535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.530460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:02.530535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.530498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:02.530733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:02.530460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:02.530733 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.530573    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:02.530733 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.530673    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:02.530733 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:02.530726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:04.530645 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:04.530612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:04.531098 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:04.530705    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:04.531098 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:04.530786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:04.531098 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:04.530888    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:04.531098 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:04.530936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:04.531098 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:04.531003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:06.530443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.530235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:06.531424 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:06.530547    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:06.531424 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.530244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:06.531424 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:06.530820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:06.531424 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.530239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:06.531424 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:06.530918    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:06.668227 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.668194    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd25bbcec769cf1adcfbe899ed035566" containerID="5aebce0231e1858971b3d8647a46df3f7cdb31ba158cc2f880519b38450a2b3d" exitCode=0
Sep 08 13:17:06.668383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.668275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"cd25bbcec769cf1adcfbe899ed035566","Type":"ContainerDied","Data":"5aebce0231e1858971b3d8647a46df3f7cdb31ba158cc2f880519b38450a2b3d"}
Sep 08 13:17:06.670129 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.670103    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="98644b34d655626b340bdf057c312dfe1894ea5aa221c0e8de0845f59952aba5" exitCode=0
Sep 08 13:17:06.670253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.670202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"98644b34d655626b340bdf057c312dfe1894ea5aa221c0e8de0845f59952aba5"}
Sep 08 13:17:06.671874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.671851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9" event={"ID":"58afe534-f025-4185-a57b-208bd1c6312a","Type":"ContainerStarted","Data":"a63b4c089f86cb708d17b336283b873272a218d3b4bd501fd5ae27912026827b"}
Sep 08 13:17:06.673478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.673456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2d8d5" event={"ID":"cc1b9c76-e22c-49f3-858f-99ffaa379061","Type":"ContainerStarted","Data":"34da3a8e840f9d65b881f8979777059c83904640cfdf4331ee00c1929a3232cc"}
Sep 08 13:17:06.674970 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.674944    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p97k" event={"ID":"228911e2-1e89-4968-9b8f-bb2d6f597d2e","Type":"ContainerStarted","Data":"afdfb3129a6dad6de1d47d6e44f5a2019adaad44a8882e6e90cbee9edbd2746f"}
Sep 08 13:17:06.677773 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.677758    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:17:06.678056 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678040    2580 generic.go:356] "Generic (PLEG): container finished" podID="bb50200c-d163-4934-9370-389b297bf6b3" containerID="24117fcd8fe2adaf6e938f27500ea6c69669116cfbbb5b94dbe0747f0a9ee39d" exitCode=1
Sep 08 13:17:06.678123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"e34bafc15231681385334d646286c5b23315f387295aefe84bf6fbcc9af1273c"}
Sep 08 13:17:06.678178 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"6628f29b4b7eb0c6ee56c6f7d856cbb0cb8c955a48a5f935980efbad631e220a"}
Sep 08 13:17:06.678223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"ecabef4e31d02bdc78156af2f9e2230be371876237fa48192425fce7bf0e8368"}
Sep 08 13:17:06.678262 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"4604de41afaeb5f78d704e6233dcc9be2527fd59e178cc61d1e23b400f6880e3"}
Sep 08 13:17:06.678298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerDied","Data":"24117fcd8fe2adaf6e938f27500ea6c69669116cfbbb5b94dbe0747f0a9ee39d"}
Sep 08 13:17:06.678331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.678307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"c639b569fa5145a6142cc9395e9a59960915019b4acdb26c24b13aaf1b32f728"}
Sep 08 13:17:06.679510 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.679489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"b7691c42ef3bfce427afbddd33d4f4b3","Type":"ContainerStarted","Data":"5d68e637a82d4c649cd9dcfd3b3c30c99306c7295fa3833cf5eaf81b0b1500cc"}
Sep 08 13:17:06.680961 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.680942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7sl66" event={"ID":"272ac4cc-2f4d-45de-b56b-e83b1918116c","Type":"ContainerStarted","Data":"2858d6878eecd8f0da4ef209412f9f6cc3812c1b390ba4950cbd8966b442192b"}
Sep 08 13:17:06.682412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.682387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x52hp" event={"ID":"684f95d3-ca3e-45fe-8dad-9487ee06e49d","Type":"ContainerStarted","Data":"a95a4a7f914ed07038bb70294d472281206b7a94c48c5f7d863ecb3624a0cbf9"}
Sep 08 13:17:06.683740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.683721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkmcw" event={"ID":"f3af11ec-f900-46e3-a79d-c54f27ef55f4","Type":"ContainerStarted","Data":"4a8698a4e556b24cc238ee6fd2de71522ea81d54dabffc2f760ad706ce75db98"}
Sep 08 13:17:06.701483 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.701447    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5p97k" podStartSLOduration=2.913998252 podStartE2EDuration="21.701434621s" totalImagesPullingTime="18.787436369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.701078403 +0000 UTC m=+22.651732354" watchObservedRunningTime="2026-09-08 13:17:06.701434621 +0000 UTC m=+22.652088573"
Sep 08 13:17:06.717422 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.717380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7sl66" podStartSLOduration=2.930053674 podStartE2EDuration="21.71737168s" totalImagesPullingTime="18.787318006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.717363912 +0000 UTC m=+22.668017864" watchObservedRunningTime="2026-09-08 13:17:06.71737168 +0000 UTC m=+22.668025631"
Sep 08 13:17:06.732291 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.732248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" podStartSLOduration=20.732232442 podStartE2EDuration="20.732232442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.731410894 +0000 UTC m=+22.682064847" watchObservedRunningTime="2026-09-08 13:17:06.732232442 +0000 UTC m=+22.682886395"
Sep 08 13:17:06.768520 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.768481    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x52hp" podStartSLOduration=2.235298906 podStartE2EDuration="21.768468751s" totalImagesPullingTime="19.533169845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.768266382 +0000 UTC m=+22.718920333" watchObservedRunningTime="2026-09-08 13:17:06.768468751 +0000 UTC m=+22.719122702"
Sep 08 13:17:06.768733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.768715    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mkmcw" podStartSLOduration=2.942258905 podStartE2EDuration="21.768708963s" totalImagesPullingTime="18.826450058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.750006906 +0000 UTC m=+22.700660858" watchObservedRunningTime="2026-09-08 13:17:06.768708963 +0000 UTC m=+22.719362915"
Sep 08 13:17:06.786615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:06.786523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2d8d5" podStartSLOduration=3.0186833809999998 podStartE2EDuration="21.786505753s" totalImagesPullingTime="18.767822372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.786134405 +0000 UTC m=+22.736788358" watchObservedRunningTime="2026-09-08 13:17:06.786505753 +0000 UTC m=+22.737159705"
Sep 08 13:17:07.687635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:07.687601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"cd25bbcec769cf1adcfbe899ed035566","Type":"ContainerStarted","Data":"7c78db1e91a2ccec3d8bc1781289ecbd33678f328394784c3bb8a381df960471"}
Sep 08 13:17:07.689142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:07.689118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqt4r" event={"ID":"5fcd4a12-9080-4095-9e1f-28c1107d6de5","Type":"ContainerStarted","Data":"b5574f508ae351236c7e9fc861b26c920411ac5f0f82c772397d5ffa88d4da14"}
Sep 08 13:17:07.704909 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:07.704875    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" podStartSLOduration=21.704863827 podStartE2EDuration="21.704863827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:07.704437883 +0000 UTC m=+23.655091837" watchObservedRunningTime="2026-09-08 13:17:07.704863827 +0000 UTC m=+23.655517780"
Sep 08 13:17:07.720681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:07.720643    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fqt4r" podStartSLOduration=3.923865793 podStartE2EDuration="22.720630937s" totalImagesPullingTime="18.796765144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:07.720308558 +0000 UTC m=+23.670962511" watchObservedRunningTime="2026-09-08 13:17:07.720630937 +0000 UTC m=+23.671284889"
Sep 08 13:17:07.811179 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:07.811140    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:17:08.468728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.468629    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:17:07.811158144Z","UUID":"7d5e3af7-2175-4f88-bbb5-5f9c65797ee5","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:17:08.470247 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.470222    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:17:08.470383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.470260    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:17:08.529848 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.529819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:08.530022 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.529819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:08.530022 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:08.529933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:08.530022 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.529947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:08.530221 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:08.530056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:08.530221 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:08.530143    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:08.696915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.696677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9" event={"ID":"58afe534-f025-4185-a57b-208bd1c6312a","Type":"ContainerStarted","Data":"177a7682c9d799ac2d4ba23409bd72d4cbe7a830cd6b98a5aae9c205753886a7"}
Sep 08 13:17:08.699758 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.699740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:17:08.876276 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.876158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:17:08.876765 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:08.876739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:17:09.706723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:09.706694    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:17:09.707205 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:09.707180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"b102a8cd4d3cd061e5493a63ebcf7ec47b52ca22ac4f6cb759c576654e639d05"}
Sep 08 13:17:09.707534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:09.707511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:17:09.708028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:09.708010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mkmcw"
Sep 08 13:17:10.530322 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:10.530266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:10.530322 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:10.530301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:10.530548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:10.530268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:10.530548 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:10.530402    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:10.530548 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:10.530486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:10.530692 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:10.530567    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:12.530665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.530466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:12.531131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.530463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:12.531131 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:12.530774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:12.531131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.530496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:12.531131 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:12.530851    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:12.531131 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:12.530913    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:12.714509 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.714479    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="458e968e88079268c9bb30d72b9f12d21b42e4b20508b1d232d99315a3b1afe0" exitCode=0
Sep 08 13:17:12.714659 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.714569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"458e968e88079268c9bb30d72b9f12d21b42e4b20508b1d232d99315a3b1afe0"}
Sep 08 13:17:12.718206 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.718187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:17:12.718568 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.718548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"b6ffc00d2e6bf02f7b1f559f2607a0b6a5100fc4e8da87ae23abb541f4d74280"}
Sep 08 13:17:12.718864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.718845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:12.718958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.718941    2580 scope.go:117] "RemoveContainer" containerID="24117fcd8fe2adaf6e938f27500ea6c69669116cfbbb5b94dbe0747f0a9ee39d"
Sep 08 13:17:12.735915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:12.735896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:13.724114 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.724078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9" event={"ID":"58afe534-f025-4185-a57b-208bd1c6312a","Type":"ContainerStarted","Data":"5e0bfdcd2dcdba04963cc5925d2a5dda565c8d008e92870f4e9316a9515d572d"}
Sep 08 13:17:13.727752 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.727733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:17:13.728081 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.728063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" event={"ID":"bb50200c-d163-4934-9370-389b297bf6b3","Type":"ContainerStarted","Data":"b7c443951c1e607b3242e8dab378e5e393c1b3c5b59d5fb32d959382a8bbdc27"}
Sep 08 13:17:13.728357 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.728336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:13.728462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.728397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:13.743812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.743635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:13.743959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.743832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hg5z9" podStartSLOduration=3.780788544 podStartE2EDuration="28.743814362s" totalImagesPullingTime="24.963025818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:13.743221232 +0000 UTC m=+29.693875186" watchObservedRunningTime="2026-09-08 13:17:13.743814362 +0000 UTC m=+29.694468316"
Sep 08 13:17:13.771673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.771633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb" podStartSLOduration=9.763998813 podStartE2EDuration="28.771619019s" totalImagesPullingTime="19.007620206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:13.771383636 +0000 UTC m=+29.722037600" watchObservedRunningTime="2026-09-08 13:17:13.771619019 +0000 UTC m=+29.722272971"
Sep 08 13:17:13.838509 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.838472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q54b"]
Sep 08 13:17:13.838669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.838652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:13.838816 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:13.838777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:13.841988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.841960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fv5ww"]
Sep 08 13:17:13.842109 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.842065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:13.842211 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:13.842185    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:13.842615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.842597    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cbrp"]
Sep 08 13:17:13.842682 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:13.842670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:13.842776 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:13.842760    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:14.733955 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:14.733920    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="6df8e4a842ec6ae82491be6a6d78a10c929f20221e092748c0b7559dced9f001" exitCode=0
Sep 08 13:17:14.734414 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:14.734002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"6df8e4a842ec6ae82491be6a6d78a10c929f20221e092748c0b7559dced9f001"}
Sep 08 13:17:15.530587 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:15.530554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:15.530733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:15.530600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:15.530733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:15.530654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:15.530733 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:15.530670    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:15.530858 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:15.530747    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:15.530858 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:15.530843    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:15.738423 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:15.738389    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="9f9cdc75d8be88644d5246a0ede1c810cf87a8d8cdcf473769123201ea0a0c89" exitCode=0
Sep 08 13:17:15.738926 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:15.738474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"9f9cdc75d8be88644d5246a0ede1c810cf87a8d8cdcf473769123201ea0a0c89"}
Sep 08 13:17:17.530193 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:17.530144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:17.530193 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:17.530180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:17.530613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:17.530179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:17.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:17.530313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:17:17.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:17.530444    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cbrp" podUID="c09eb737-001f-4c00-bd76-492f842e8980"
Sep 08 13:17:17.530613 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:17.530537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fv5ww" podUID="63cd3528-9abe-4b26-8744-c662668b3f60"
Sep 08 13:17:18.269983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.269938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.269996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.270038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270141    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270177    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270187    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270207    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:17:18.270205 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270216    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f7wr5 for pod openshift-network-diagnostics/network-check-target-8cbrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:18.270614 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.270224831 +0000 UTC m=+66.220878772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:18.270614 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret podName:63cd3528-9abe-4b26-8744-c662668b3f60 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.270256353 +0000 UTC m=+66.220910284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret") pod "global-pull-secret-syncer-fv5ww" (UID: "63cd3528-9abe-4b26-8744-c662668b3f60") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:18.270614 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:18.270282    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5 podName:c09eb737-001f-4c00-bd76-492f842e8980 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.270273928 +0000 UTC m=+66.220927873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7wr5" (UniqueName: "kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5") pod "network-check-target-8cbrp" (UID: "c09eb737-001f-4c00-bd76-492f842e8980") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:18.788739 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.788716    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeReady"
Sep 08 13:17:18.789156 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.788846    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:17:18.839358 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.839318    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6xdwz"]
Sep 08 13:17:18.882211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.882159    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrgl"]
Sep 08 13:17:18.882358 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.882290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:18.885247 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.885220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:17:18.885247 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.885237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kwlvq\""
Sep 08 13:17:18.885529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.885508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:17:18.899482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.899461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xdwz"]
Sep 08 13:17:18.899594 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.899501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrgl"]
Sep 08 13:17:18.899655 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.899603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:18.902402 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.902378    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:17:18.902548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.902524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t98tn\""
Sep 08 13:17:18.902626 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.902568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:17:18.902715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.902682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:17:18.974924 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.974853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:18.975061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.974924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96d2294-11ce-403e-9f7f-33d0106bc05e-tmp-dir\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:18.975061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.974962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d2294-11ce-403e-9f7f-33d0106bc05e-config-volume\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:18.975061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.974981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmmx\" (UniqueName: \"kubernetes.io/projected/a96d2294-11ce-403e-9f7f-33d0106bc05e-kube-api-access-ldmmx\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:18.975061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.975007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d95a0cd-2074-4446-97db-45ae6a01e3bf-cert\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:18.975061 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:18.975052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zs6b\" (UniqueName: \"kubernetes.io/projected/2d95a0cd-2074-4446-97db-45ae6a01e3bf-kube-api-access-7zs6b\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.075751 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.075717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zs6b\" (UniqueName: \"kubernetes.io/projected/2d95a0cd-2074-4446-97db-45ae6a01e3bf-kube-api-access-7zs6b\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.075751 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.075765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.075977 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:19.075866    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:19.075977 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.075911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96d2294-11ce-403e-9f7f-33d0106bc05e-tmp-dir\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.075977 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:19.075932    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:19.575912531 +0000 UTC m=+35.526566477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:19.075977 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.075974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d2294-11ce-403e-9f7f-33d0106bc05e-config-volume\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.076196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.075998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmmx\" (UniqueName: \"kubernetes.io/projected/a96d2294-11ce-403e-9f7f-33d0106bc05e-kube-api-access-ldmmx\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.076196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.076021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d95a0cd-2074-4446-97db-45ae6a01e3bf-cert\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.076299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.076230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96d2294-11ce-403e-9f7f-33d0106bc05e-tmp-dir\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.076642 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.076618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a96d2294-11ce-403e-9f7f-33d0106bc05e-config-volume\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.081376 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.081221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d95a0cd-2074-4446-97db-45ae6a01e3bf-cert\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.086310 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.086283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zs6b\" (UniqueName: \"kubernetes.io/projected/2d95a0cd-2074-4446-97db-45ae6a01e3bf-kube-api-access-7zs6b\") pod \"ingress-canary-5hrgl\" (UID: \"2d95a0cd-2074-4446-97db-45ae6a01e3bf\") " pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.086663 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.086640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmmx\" (UniqueName: \"kubernetes.io/projected/a96d2294-11ce-403e-9f7f-33d0106bc05e-kube-api-access-ldmmx\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.209280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.209246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hrgl"
Sep 08 13:17:19.530840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.530756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:19.531016 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.530756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:19.531016 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.530757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:19.533936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.533912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:19.534074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.533944    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxg9n\""
Sep 08 13:17:19.534074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.533913    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:19.534074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.534035    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:19.535018 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.534992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:19.535116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.535065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g794g\""
Sep 08 13:17:19.580083 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:19.580049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:19.580252 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:19.580206    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:19.580335 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:19.580323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:20.580300503 +0000 UTC m=+36.530954456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:20.187532 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:20.187491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrgl"]
Sep 08 13:17:20.192055 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:17:20.192031    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d95a0cd_2074_4446_97db_45ae6a01e3bf.slice/crio-10a6274733f64ca8305db7a088f8eef7eff3aa2cf6e42e9d993154d11acefa02 WatchSource:0}: Error finding container 10a6274733f64ca8305db7a088f8eef7eff3aa2cf6e42e9d993154d11acefa02: Status 404 returned error can't find the container with id 10a6274733f64ca8305db7a088f8eef7eff3aa2cf6e42e9d993154d11acefa02
Sep 08 13:17:20.589418 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:20.589382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:20.589597 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:20.589539    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:20.589648 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:20.589598    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:22.5895845 +0000 UTC m=+38.540238430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:20.751296 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:20.751261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hrgl" event={"ID":"2d95a0cd-2074-4446-97db-45ae6a01e3bf","Type":"ContainerStarted","Data":"10a6274733f64ca8305db7a088f8eef7eff3aa2cf6e42e9d993154d11acefa02"}
Sep 08 13:17:20.754014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:20.753981    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="903717fd3a824915ca114d7220d6c76fe88322ebd6a0b427fdd0f13e5097e994" exitCode=0
Sep 08 13:17:20.754123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:20.754029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"903717fd3a824915ca114d7220d6c76fe88322ebd6a0b427fdd0f13e5097e994"}
Sep 08 13:17:21.759559 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:21.759518    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcabf8af-b206-4d16-b396-8a875b739614" containerID="f629267692e42757b6ee256313d37802b8b2a568f49f834f8e6ccfa7e9bae368" exitCode=0
Sep 08 13:17:21.760324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:21.759587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerDied","Data":"f629267692e42757b6ee256313d37802b8b2a568f49f834f8e6ccfa7e9bae368"}
Sep 08 13:17:22.606577 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:22.606426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:22.606665 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:22.606578    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:22.606665 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:22.606650    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:26.606628863 +0000 UTC m=+42.557282795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:22.764718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:22.764678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfl22" event={"ID":"dcabf8af-b206-4d16-b396-8a875b739614","Type":"ContainerStarted","Data":"ef6744d593ded1bb73b94312831753f77a05abbdc7bbe9730eb42531e15e91da"}
Sep 08 13:17:22.790251 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:22.790145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mfl22" podStartSLOduration=6.783518463 podStartE2EDuration="37.790130294s" totalImagesPullingTime="31.006611831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:22.788520398 +0000 UTC m=+38.739174349" watchObservedRunningTime="2026-09-08 13:17:22.790130294 +0000 UTC m=+38.740784277"
Sep 08 13:17:23.768246 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:23.768210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hrgl" event={"ID":"2d95a0cd-2074-4446-97db-45ae6a01e3bf","Type":"ContainerStarted","Data":"0ad766acf35e4d9d65d7d62ea0fd07721eea12f1b1ca0e00ddd51a8ba6f2334d"}
Sep 08 13:17:23.785420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:23.785380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5hrgl" podStartSLOduration=3.195420549 podStartE2EDuration="5.785365832s" totalImagesPullingTime="2.589945283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:23.784278368 +0000 UTC m=+39.734932319" watchObservedRunningTime="2026-09-08 13:17:23.785365832 +0000 UTC m=+39.736019800"
Sep 08 13:17:26.636303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:26.636264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:26.636685 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:26.636379    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:26.636685 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:26.636428    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:34.636416299 +0000 UTC m=+50.587070229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:34.688990 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:34.688946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:34.689485 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:34.689091    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:34.689485 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:34.689159    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.689142679 +0000 UTC m=+66.639796609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:45.748142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:45.748115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv5hb"
Sep 08 13:17:50.288056 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.288013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:50.288056 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.288058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:17:50.288520 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.288095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:50.291047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.291023    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:50.291185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.291137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:50.291185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.291143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:50.298525 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:50.298507    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:50.298584 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:50.298561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.298546462 +0000 UTC m=+130.249200392 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : secret "metrics-daemon-secret" not found
Sep 08 13:17:50.301192 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.301153    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:50.301561 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.301543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cd3528-9abe-4b26-8744-c662668b3f60-original-pull-secret\") pod \"global-pull-secret-syncer-fv5ww\" (UID: \"63cd3528-9abe-4b26-8744-c662668b3f60\") " pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:50.310940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.310920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wr5\" (UniqueName: \"kubernetes.io/projected/c09eb737-001f-4c00-bd76-492f842e8980-kube-api-access-f7wr5\") pod \"network-check-target-8cbrp\" (UID: \"c09eb737-001f-4c00-bd76-492f842e8980\") " pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:50.443503 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.443466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fv5ww"
Sep 08 13:17:50.453423 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.453399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g794g\""
Sep 08 13:17:50.461647 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.461618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:50.690575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.690541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:17:50.690750 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:50.690679    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:50.690750 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:17:50.690736    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:18:22.690721299 +0000 UTC m=+98.641375234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:50.745875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.745846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fv5ww"]
Sep 08 13:17:50.749128 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:17:50.749088    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cd3528_9abe_4b26_8744_c662668b3f60.slice/crio-fcd3b2ba8284d10dd83956636c53891088ca6e0330eb3a22a2b5bf3927a1e6e4 WatchSource:0}: Error finding container fcd3b2ba8284d10dd83956636c53891088ca6e0330eb3a22a2b5bf3927a1e6e4: Status 404 returned error can't find the container with id fcd3b2ba8284d10dd83956636c53891088ca6e0330eb3a22a2b5bf3927a1e6e4
Sep 08 13:17:50.768720 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.768697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cbrp"]
Sep 08 13:17:50.771546 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:17:50.771513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09eb737_001f_4c00_bd76_492f842e8980.slice/crio-69ff21b4219f3d96ebfd3d77d53a407f78fa376d79fa3dd96fee5c7a8adf935b WatchSource:0}: Error finding container 69ff21b4219f3d96ebfd3d77d53a407f78fa376d79fa3dd96fee5c7a8adf935b: Status 404 returned error can't find the container with id 69ff21b4219f3d96ebfd3d77d53a407f78fa376d79fa3dd96fee5c7a8adf935b
Sep 08 13:17:50.832663 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.832586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cbrp" event={"ID":"c09eb737-001f-4c00-bd76-492f842e8980","Type":"ContainerStarted","Data":"69ff21b4219f3d96ebfd3d77d53a407f78fa376d79fa3dd96fee5c7a8adf935b"}
Sep 08 13:17:50.833772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:50.833751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fv5ww" event={"ID":"63cd3528-9abe-4b26-8744-c662668b3f60","Type":"ContainerStarted","Data":"fcd3b2ba8284d10dd83956636c53891088ca6e0330eb3a22a2b5bf3927a1e6e4"}
Sep 08 13:17:56.851504 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:56.851466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cbrp" event={"ID":"c09eb737-001f-4c00-bd76-492f842e8980","Type":"ContainerStarted","Data":"b0e18f6e1abe9e20ec426922f940586f14d8120f306492b2eac0e836ac7f5ef9"}
Sep 08 13:17:56.851971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:56.851599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:17:56.853028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:56.853007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fv5ww" event={"ID":"63cd3528-9abe-4b26-8744-c662668b3f60","Type":"ContainerStarted","Data":"f0e2117de94f7df04345f5c699a70b5d28240208717b9d78ab4fb99465e828eb"}
Sep 08 13:17:56.868367 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:56.868325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8cbrp" podStartSLOduration=66.512405681 podStartE2EDuration="1m11.868312834s" totalImagesPullingTime="5.355907153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:56.867586792 +0000 UTC m=+72.818240743" watchObservedRunningTime="2026-09-08 13:17:56.868312834 +0000 UTC m=+72.818966780"
Sep 08 13:17:56.883032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:17:56.882996    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fv5ww" podStartSLOduration=66.501142345 podStartE2EDuration="1m11.882985598s" totalImagesPullingTime="5.381843253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:56.882532165 +0000 UTC m=+72.833186118" watchObservedRunningTime="2026-09-08 13:17:56.882985598 +0000 UTC m=+72.833639550"
Sep 08 13:18:22.703409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:22.703268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:18:22.703878 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:22.703430    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:18:22.703878 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:22.703527    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls podName:a96d2294-11ce-403e-9f7f-33d0106bc05e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:26.703507879 +0000 UTC m=+162.654161823 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls") pod "dns-default-6xdwz" (UID: "a96d2294-11ce-403e-9f7f-33d0106bc05e") : secret "dns-default-metrics-tls" not found
Sep 08 13:18:27.857804 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:27.857773    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8cbrp"
Sep 08 13:18:54.013779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.013743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k5hjh"]
Sep 08 13:18:54.015484 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.015467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh"
Sep 08 13:18:54.019105 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.019086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t8brp\""
Sep 08 13:18:54.022315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.022296    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:18:54.023900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.023882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.028699 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.028680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k5hjh"]
Sep 08 13:18:54.029596 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.029534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:18:54.030181 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.030147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:18:54.030181 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.030155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:18:54.030837 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.030821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55db4\""
Sep 08 13:18:54.038330 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.038304    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:18:54.044835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.044814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:18:54.101419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr762\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgj8\" (UniqueName: \"kubernetes.io/projected/c10475ca-573e-4e8c-b618-c0f761e5e1ac-kube-api-access-twgj8\") pod \"network-check-source-685456655f-k5hjh\" (UID: \"c10475ca-573e-4e8c-b618-c0f761e5e1ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh"
Sep 08 13:18:54.101687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.101687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.101640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.110608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.110584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"]
Sep 08 13:18:54.112282 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.112266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.114865 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.114827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 13:18:54.114865 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.114846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.114985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.114921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gz2j2\""
Sep 08 13:18:54.114985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.114979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.115057 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.115045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 13:18:54.117289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.117267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"]
Sep 08 13:18:54.119028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.119009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.119226 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.119211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qhdh8"]
Sep 08 13:18:54.120745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.120727    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"]
Sep 08 13:18:54.120869 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.120851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.122637 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.122617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.122732 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.122669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 13:18:54.123042 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kcbmj\""
Sep 08 13:18:54.123264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.123373 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 13:18:54.123602 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123589    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 13:18:54.123659 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2b6pd\""
Sep 08 13:18:54.123845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123828    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 13:18:54.123906 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.123893    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.125207 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.125187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dqklc\""
Sep 08 13:18:54.125716 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.125698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 13:18:54.125866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.125850    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.126111 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.126094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.126206 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.126188    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"]
Sep 08 13:18:54.133765 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.133743    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"]
Sep 08 13:18:54.134933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.134912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qhdh8"]
Sep 08 13:18:54.136351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.136332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"]
Sep 08 13:18:54.202776 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.202749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr762\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.202905 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.202786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.202905 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.202808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nt8x\" (UniqueName: \"kubernetes.io/projected/803f4dce-540e-4ff8-9387-e696f4975b8d-kube-api-access-5nt8x\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.202905 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.202854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/428db53a-2e22-4190-ab14-83725d171d16-config\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.203036 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.202993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/468d9440-90f8-4110-a67b-d6aec9a871ad-nginx-conf\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.203131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgj8\" (UniqueName: \"kubernetes.io/projected/c10475ca-573e-4e8c-b618-c0f761e5e1ac-kube-api-access-twgj8\") pod \"network-check-source-685456655f-k5hjh\" (UID: \"c10475ca-573e-4e8c-b618-c0f761e5e1ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh"
Sep 08 13:18:54.203218 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203218 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/428db53a-2e22-4190-ab14-83725d171d16-serving-cert\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.203334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.203334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.203334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26lq\" (UniqueName: \"kubernetes.io/projected/428db53a-2e22-4190-ab14-83725d171d16-kube-api-access-s26lq\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgdw\" (UniqueName: \"kubernetes.io/projected/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-kube-api-access-lzgdw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.203628    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.203645    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.203721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.703702816 +0000 UTC m=+130.654356763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:18:54.203981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.203790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.204314 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.204291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.205467 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.205451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.205685 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.205662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.209073 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.209051    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"]
Sep 08 13:18:54.215062 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.215038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"
Sep 08 13:18:54.216387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.216359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr762\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.216485 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.216450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgj8\" (UniqueName: \"kubernetes.io/projected/c10475ca-573e-4e8c-b618-c0f761e5e1ac-kube-api-access-twgj8\") pod \"network-check-source-685456655f-k5hjh\" (UID: \"c10475ca-573e-4e8c-b618-c0f761e5e1ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh"
Sep 08 13:18:54.216617 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.216598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.217651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.217631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-twb4c\""
Sep 08 13:18:54.217743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.217665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.217914 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.217899    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.223778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.223759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"]
Sep 08 13:18:54.304972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.304906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nt8x\" (UniqueName: \"kubernetes.io/projected/803f4dce-540e-4ff8-9387-e696f4975b8d-kube-api-access-5nt8x\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.304972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.304937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/428db53a-2e22-4190-ab14-83725d171d16-config\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.304972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.304954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/468d9440-90f8-4110-a67b-d6aec9a871ad-nginx-conf\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.304985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/428db53a-2e22-4190-ab14-83725d171d16-serving-cert\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26lq\" (UniqueName: \"kubernetes.io/projected/428db53a-2e22-4190-ab14-83725d171d16-kube-api-access-s26lq\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pttr\" (UniqueName: \"kubernetes.io/projected/ae6234bd-edff-4975-9758-f5780c517a8b-kube-api-access-8pttr\") pod \"volume-data-source-validator-695c895bd6-zt762\" (UID: \"ae6234bd-edff-4975-9758-f5780c517a8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305119    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305119    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.305236 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls podName:803f4dce-540e-4ff8-9387-e696f4975b8d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.805204091 +0000 UTC m=+130.755858038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bxmwf" (UID: "803f4dce-540e-4ff8-9387-e696f4975b8d") : secret "samples-operator-tls" not found
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305257    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.805235732 +0000 UTC m=+130.755889673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgdw\" (UniqueName: \"kubernetes.io/projected/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-kube-api-access-lzgdw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305465    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.305505    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs podName:05fa069d-7b89-4ce6-823f-613bfb31539a nodeName:}" failed. No retries permitted until 2026-09-08 13:20:56.305491857 +0000 UTC m=+252.256145801 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs") pod "network-metrics-daemon-4q54b" (UID: "05fa069d-7b89-4ce6-823f-613bfb31539a") : secret "metrics-daemon-secret" not found
Sep 08 13:18:54.305625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/428db53a-2e22-4190-ab14-83725d171d16-config\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.305843 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/468d9440-90f8-4110-a67b-d6aec9a871ad-nginx-conf\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.305879 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.305840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.307250 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.307230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.307331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.307313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/428db53a-2e22-4190-ab14-83725d171d16-serving-cert\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.314343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.314317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nt8x\" (UniqueName: \"kubernetes.io/projected/803f4dce-540e-4ff8-9387-e696f4975b8d-kube-api-access-5nt8x\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.314343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.314321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgdw\" (UniqueName: \"kubernetes.io/projected/ea5563ed-b472-4c08-8ecc-f61b5cd573ee-kube-api-access-lzgdw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-qnzzr\" (UID: \"ea5563ed-b472-4c08-8ecc-f61b5cd573ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.314459 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.314387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26lq\" (UniqueName: \"kubernetes.io/projected/428db53a-2e22-4190-ab14-83725d171d16-kube-api-access-s26lq\") pod \"service-ca-operator-859f94c4b-29zz9\" (UID: \"428db53a-2e22-4190-ab14-83725d171d16\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.324429 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.324414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh"
Sep 08 13:18:54.405737 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.405708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pttr\" (UniqueName: \"kubernetes.io/projected/ae6234bd-edff-4975-9758-f5780c517a8b-kube-api-access-8pttr\") pod \"volume-data-source-validator-695c895bd6-zt762\" (UID: \"ae6234bd-edff-4975-9758-f5780c517a8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"
Sep 08 13:18:54.413951 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.413925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pttr\" (UniqueName: \"kubernetes.io/projected/ae6234bd-edff-4975-9758-f5780c517a8b-kube-api-access-8pttr\") pod \"volume-data-source-validator-695c895bd6-zt762\" (UID: \"ae6234bd-edff-4975-9758-f5780c517a8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"
Sep 08 13:18:54.421696 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.421675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"
Sep 08 13:18:54.430650 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.430628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"
Sep 08 13:18:54.530715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.530673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"
Sep 08 13:18:54.590254 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.590220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k5hjh"]
Sep 08 13:18:54.593128 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:18:54.593097    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10475ca_573e_4e8c_b618_c0f761e5e1ac.slice/crio-9ab61f94fe35b635d02c7579e984c7e44cf8a9aa9bd8089321e117d825dd46e7 WatchSource:0}: Error finding container 9ab61f94fe35b635d02c7579e984c7e44cf8a9aa9bd8089321e117d825dd46e7: Status 404 returned error can't find the container with id 9ab61f94fe35b635d02c7579e984c7e44cf8a9aa9bd8089321e117d825dd46e7
Sep 08 13:18:54.691335 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.691303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9"]
Sep 08 13:18:54.694660 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:18:54.694632    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428db53a_2e22_4190_ab14_83725d171d16.slice/crio-28e57598af1a18701ccfab028f74e1e56e931b5b129f7fc012277e3cc8065043 WatchSource:0}: Error finding container 28e57598af1a18701ccfab028f74e1e56e931b5b129f7fc012277e3cc8065043: Status 404 returned error can't find the container with id 28e57598af1a18701ccfab028f74e1e56e931b5b129f7fc012277e3cc8065043
Sep 08 13:18:54.708307 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.708285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:54.708441 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.708423    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:54.708497 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.708442    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:18:54.708497 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.708489    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.708474846 +0000 UTC m=+131.659128776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:18:54.718567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.718542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr"]
Sep 08 13:18:54.721464 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:18:54.721443    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5563ed_b472_4c08_8ecc_f61b5cd573ee.slice/crio-dbecb4c9d9b42e97beee7201b13129d83efd97bacd9eddb2ecd0125a1d2f51dc WatchSource:0}: Error finding container dbecb4c9d9b42e97beee7201b13129d83efd97bacd9eddb2ecd0125a1d2f51dc: Status 404 returned error can't find the container with id dbecb4c9d9b42e97beee7201b13129d83efd97bacd9eddb2ecd0125a1d2f51dc
Sep 08 13:18:54.796145 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.796122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762"]
Sep 08 13:18:54.798921 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:18:54.798898    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6234bd_edff_4975_9758_f5780c517a8b.slice/crio-4d2bc2b5e2690cc4a0de5732496fd070254d0d9b09faae7d262509077edfecf6 WatchSource:0}: Error finding container 4d2bc2b5e2690cc4a0de5732496fd070254d0d9b09faae7d262509077edfecf6: Status 404 returned error can't find the container with id 4d2bc2b5e2690cc4a0de5732496fd070254d0d9b09faae7d262509077edfecf6
Sep 08 13:18:54.809339 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.809320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:54.809421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.809349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:54.809478 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.809461    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:54.809524 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.809514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.809499204 +0000 UTC m=+131.760153134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:54.809524 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.809516    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:54.809609 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:54.809543    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls podName:803f4dce-540e-4ff8-9387-e696f4975b8d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.809536942 +0000 UTC m=+131.760190873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bxmwf" (UID: "803f4dce-540e-4ff8-9387-e696f4975b8d") : secret "samples-operator-tls" not found
Sep 08 13:18:54.991134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.991103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh" event={"ID":"c10475ca-573e-4e8c-b618-c0f761e5e1ac","Type":"ContainerStarted","Data":"6b47d1815f5724f48dc1ffbf3ea41a40f58d63e2aefb31e8addea7513b9e0dfd"}
Sep 08 13:18:54.991324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.991151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh" event={"ID":"c10475ca-573e-4e8c-b618-c0f761e5e1ac","Type":"ContainerStarted","Data":"9ab61f94fe35b635d02c7579e984c7e44cf8a9aa9bd8089321e117d825dd46e7"}
Sep 08 13:18:54.992418 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.992397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762" event={"ID":"ae6234bd-edff-4975-9758-f5780c517a8b","Type":"ContainerStarted","Data":"4d2bc2b5e2690cc4a0de5732496fd070254d0d9b09faae7d262509077edfecf6"}
Sep 08 13:18:54.993579 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.993554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr" event={"ID":"ea5563ed-b472-4c08-8ecc-f61b5cd573ee","Type":"ContainerStarted","Data":"dbecb4c9d9b42e97beee7201b13129d83efd97bacd9eddb2ecd0125a1d2f51dc"}
Sep 08 13:18:54.994597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:54.994577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9" event={"ID":"428db53a-2e22-4190-ab14-83725d171d16","Type":"ContainerStarted","Data":"28e57598af1a18701ccfab028f74e1e56e931b5b129f7fc012277e3cc8065043"}
Sep 08 13:18:55.008492 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:55.008453    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-k5hjh" podStartSLOduration=2.008441046 podStartE2EDuration="2.008441046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:55.007757887 +0000 UTC m=+130.958411855" watchObservedRunningTime="2026-09-08 13:18:55.008441046 +0000 UTC m=+130.959094999"
Sep 08 13:18:55.718106 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:55.718064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:55.718588 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.718231    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:55.718588 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.718246    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:18:55.718588 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.718294    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.718278471 +0000 UTC m=+133.668932401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:18:55.819147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:55.819110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:55.819324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:55.819160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:55.819324 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.819278    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:55.819411 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.819339    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls podName:803f4dce-540e-4ff8-9387-e696f4975b8d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.819321755 +0000 UTC m=+133.769975690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bxmwf" (UID: "803f4dce-540e-4ff8-9387-e696f4975b8d") : secret "samples-operator-tls" not found
Sep 08 13:18:55.819411 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.819278    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:55.819513 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:55.819418    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.819400989 +0000 UTC m=+133.770054924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:57.000901 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.000817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr" event={"ID":"ea5563ed-b472-4c08-8ecc-f61b5cd573ee","Type":"ContainerStarted","Data":"a0324ac8c4ce200f5bbc9ebfd02bf0f60bd2a9e0157e03e81c05308799ceb608"}
Sep 08 13:18:57.018437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.018393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr" podStartSLOduration=1.019326445 podStartE2EDuration="3.018378747s" totalImagesPullingTime="1.999052302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:57.017181046 +0000 UTC m=+132.967834992" watchObservedRunningTime="2026-09-08 13:18:57.018378747 +0000 UTC m=+132.969032699"
Sep 08 13:18:57.667308 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.667273    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7ls8l"]
Sep 08 13:18:57.669744 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.669725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.672236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.672214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:18:57.673415 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.673392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:18:57.673415 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.673408    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:18:57.673561 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.673409    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kpczc\""
Sep 08 13:18:57.673561 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.673408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:18:57.678965 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.678946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7ls8l"]
Sep 08 13:18:57.734813 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdkk\" (UniqueName: \"kubernetes.io/projected/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-api-access-dxdkk\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.734971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-crio-socket\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.734971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-data-volume\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.734971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.734971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.735102 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.734985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:18:57.735102 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.735095    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:57.735185 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.735105    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:18:57.735185 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.735159    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.735145737 +0000 UTC m=+137.685799670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:18:57.836365 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.836536 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:18:57.836536 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.836536 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:18:57.836536 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836482    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836535    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836544    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:58.336533992 +0000 UTC m=+134.287187926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdkk\" (UniqueName: \"kubernetes.io/projected/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-api-access-dxdkk\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836580    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.836569627 +0000 UTC m=+137.787223557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836540    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls podName:803f4dce-540e-4ff8-9387-e696f4975b8d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.83659343 +0000 UTC m=+137.787247359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bxmwf" (UID: "803f4dce-540e-4ff8-9387-e696f4975b8d") : secret "samples-operator-tls" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:57.836627    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:58.336619099 +0000 UTC m=+134.287273028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:57.836693 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-crio-socket\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.836991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-data-volume\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.836991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-crio-socket\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.837053 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.836991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-data-volume\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:57.844767 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:57.844742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdkk\" (UniqueName: \"kubernetes.io/projected/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-api-access-dxdkk\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:58.340769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.340721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:58.341161 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.340787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:58.341161 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:58.340871    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:58.341161 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:58.340891    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:58.341161 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:58.340943    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:59.340927077 +0000 UTC m=+135.291581007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:58.341161 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:58.340967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:59.34095347 +0000 UTC m=+135.291607400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:58.762524 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.762491    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"]
Sep 08 13:18:58.764795 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.764779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"
Sep 08 13:18:58.773026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.773005    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xn25w\""
Sep 08 13:18:58.773132 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.773029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:58.774041 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.774025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 13:18:58.779812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.779793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"]
Sep 08 13:18:58.844224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.844157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xskp\" (UniqueName: \"kubernetes.io/projected/daca9765-01e0-4542-beb8-07a507867daf-kube-api-access-7xskp\") pod \"migrator-fdbb8fdff-j6pmg\" (UID: \"daca9765-01e0-4542-beb8-07a507867daf\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"
Sep 08 13:18:58.944764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.944724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xskp\" (UniqueName: \"kubernetes.io/projected/daca9765-01e0-4542-beb8-07a507867daf-kube-api-access-7xskp\") pod \"migrator-fdbb8fdff-j6pmg\" (UID: \"daca9765-01e0-4542-beb8-07a507867daf\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"
Sep 08 13:18:58.957512 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:58.957489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xskp\" (UniqueName: \"kubernetes.io/projected/daca9765-01e0-4542-beb8-07a507867daf-kube-api-access-7xskp\") pod \"migrator-fdbb8fdff-j6pmg\" (UID: \"daca9765-01e0-4542-beb8-07a507867daf\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"
Sep 08 13:18:59.073263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:59.073187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"
Sep 08 13:18:59.344407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:59.341766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg"]
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:59.346554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:59.346630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:59.346651    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:59.346709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.346688551 +0000 UTC m=+137.297342501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:59.346740    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:59.346802 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:18:59.346788    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.346771745 +0000 UTC m=+137.297425692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:59.441007 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:18:59.440975    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7sl66_272ac4cc-2f4d-45de-b56b-e83b1918116c/dns-node-resolver/0.log"
Sep 08 13:19:00.011928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:00.011889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg" event={"ID":"daca9765-01e0-4542-beb8-07a507867daf","Type":"ContainerStarted","Data":"c81066542b1d54ec51bc0b6c6fe5b5a32bcb820b0eab987d8df1480e89923518"}
Sep 08 13:19:00.439589 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:00.439516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2d8d5_cc1b9c76-e22c-49f3-858f-99ffaa379061/node-ca/0.log"
Sep 08 13:19:01.017428 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.017393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762" event={"ID":"ae6234bd-edff-4975-9758-f5780c517a8b","Type":"ContainerStarted","Data":"02488bf44018448dc1b4f0ae7f2392462aead0064bf1bb7fbaf77f6cb6669855"}
Sep 08 13:19:01.019326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.019295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9" event={"ID":"428db53a-2e22-4190-ab14-83725d171d16","Type":"ContainerStarted","Data":"b6019556513ca69217c0ac9b9c8bfb11ce0084724ab1d4e85bd7679ca0211a05"}
Sep 08 13:19:01.035439 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.035346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-zt762" podStartSLOduration=1.05366719 podStartE2EDuration="7.035327455s" totalImagesPullingTime="5.981660265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:01.033134285 +0000 UTC m=+136.983788243" watchObservedRunningTime="2026-09-08 13:19:01.035327455 +0000 UTC m=+136.985981409"
Sep 08 13:19:01.052631 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.052580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9" podStartSLOduration=0.961593659 podStartE2EDuration="7.052560849s" totalImagesPullingTime="6.09096719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:01.051990805 +0000 UTC m=+137.002644771" watchObservedRunningTime="2026-09-08 13:19:01.052560849 +0000 UTC m=+137.003214802"
Sep 08 13:19:01.239118 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.239094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hrgl_2d95a0cd-2074-4446-97db-45ae6a01e3bf/serve-healthcheck-canary/0.log"
Sep 08 13:19:01.366381 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.366304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:01.366381 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.366371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:01.366561 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.366473    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:01.366561 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.366509    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:01.366561 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.366557    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.366533311 +0000 UTC m=+141.317187254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:01.366661 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.366579    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.366568299 +0000 UTC m=+141.317222234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:01.770330 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.770291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:01.770701 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.770443    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:01.770701 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.770462    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:19:01.770701 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.770517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.77049957 +0000 UTC m=+145.721153506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:19:01.871305 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.871275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:19:01.871489 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:01.871387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:19:01.871489 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.871428    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:01.871489 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.871488    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls podName:803f4dce-540e-4ff8-9387-e696f4975b8d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.871473441 +0000 UTC m=+145.822127372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bxmwf" (UID: "803f4dce-540e-4ff8-9387-e696f4975b8d") : secret "samples-operator-tls" not found
Sep 08 13:19:01.871637 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.871488    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:19:01.871637 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:01.871542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.871526448 +0000 UTC m=+145.822180379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:19:02.023885 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:02.023851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg" event={"ID":"daca9765-01e0-4542-beb8-07a507867daf","Type":"ContainerStarted","Data":"9ea22dc02d7d66bd8c213a28bc6610f5835e91c224808277bb214d7a97534597"}
Sep 08 13:19:02.024036 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:02.023907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg" event={"ID":"daca9765-01e0-4542-beb8-07a507867daf","Type":"ContainerStarted","Data":"e32c74dc2c760db0a109444080813dcc0759561095c26e7d378022d6d0cf6ead"}
Sep 08 13:19:02.042974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:02.042922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-j6pmg" podStartSLOduration=1.849792401 podStartE2EDuration="4.042902064s" totalImagesPullingTime="2.193109663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:02.042488462 +0000 UTC m=+137.993142415" watchObservedRunningTime="2026-09-08 13:19:02.042902064 +0000 UTC m=+137.993556019"
Sep 08 13:19:05.402620 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:05.402560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:05.403016 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:05.402681    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:05.403016 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:05.402745    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.402728648 +0000 UTC m=+149.353382582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:05.403016 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:05.402691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:05.403016 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:05.402777    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:05.403016 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:05.402841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.402827991 +0000 UTC m=+149.353481924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:09.838427 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:09.838378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:09.838827 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:09.838528    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:09.838827 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:09.838547    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5db4cddb66-4cjpn: secret "image-registry-tls" not found
Sep 08 13:19:09.838827 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:09.838602    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls podName:fa4bb584-62d0-494c-b020-a306692b56d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:25.838587528 +0000 UTC m=+161.789241458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls") pod "image-registry-5db4cddb66-4cjpn" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0") : secret "image-registry-tls" not found
Sep 08 13:19:09.939757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:09.939718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:19:09.939918 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:09.939783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:19:09.939918 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:09.939865    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:19:09.940049 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:09.939927    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert podName:468d9440-90f8-4110-a67b-d6aec9a871ad nodeName:}" failed. No retries permitted until 2026-09-08 13:19:25.939911872 +0000 UTC m=+161.890565803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-qhdh8" (UID: "468d9440-90f8-4110-a67b-d6aec9a871ad") : secret "networking-console-plugin-cert" not found
Sep 08 13:19:09.942220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:09.942196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/803f4dce-540e-4ff8-9387-e696f4975b8d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bxmwf\" (UID: \"803f4dce-540e-4ff8-9387-e696f4975b8d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:19:10.042703 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:10.042673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"
Sep 08 13:19:10.314603 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:10.314571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf"]
Sep 08 13:19:11.053063 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:11.053022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf" event={"ID":"803f4dce-540e-4ff8-9387-e696f4975b8d","Type":"ContainerStarted","Data":"9e221762c34a628dcc973d7928b303c8b4d791bda36a1d09a3202d28a6043dcd"}
Sep 08 13:19:13.061116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.061079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf" event={"ID":"803f4dce-540e-4ff8-9387-e696f4975b8d","Type":"ContainerStarted","Data":"17af007fbe0e730519b1de2f105bc9e4b0defa5bd6ea9b44f59c519ca00c6cff"}
Sep 08 13:19:13.061521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.061140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf" event={"ID":"803f4dce-540e-4ff8-9387-e696f4975b8d","Type":"ContainerStarted","Data":"a56afe7cd4d528a1b42447215b6b5f193a0026846c8f1c38d560e6c5c8dd521c"}
Sep 08 13:19:13.080209 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.080140    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bxmwf" podStartSLOduration=17.443671454 podStartE2EDuration="19.080123042s" totalImagesPullingTime="1.636451588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:13.079201151 +0000 UTC m=+149.029855105" watchObservedRunningTime="2026-09-08 13:19:13.080123042 +0000 UTC m=+149.030777036"
Sep 08 13:19:13.466055 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.466022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:13.466200 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.466083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:13.466200 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:13.466188    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:13.466288 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:13.466252    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls podName:eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:29.466237131 +0000 UTC m=+165.416891072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ls8l" (UID: "eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:13.466771 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:13.466750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:21.893398 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:21.893359    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6xdwz" podUID="a96d2294-11ce-403e-9f7f-33d0106bc05e"
Sep 08 13:19:22.090936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:22.090905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:22.556563 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:22.556518    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4q54b" podUID="05fa069d-7b89-4ce6-823f-613bfb31539a"
Sep 08 13:19:25.852378 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:25.852342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:25.854711 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:25.854688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"image-registry-5db4cddb66-4cjpn\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") " pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:25.953045 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:25.953012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:19:25.955289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:25.955269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/468d9440-90f8-4110-a67b-d6aec9a871ad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qhdh8\" (UID: \"468d9440-90f8-4110-a67b-d6aec9a871ad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:19:26.132351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.132278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:26.176268 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.176232    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:19:26.226070 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.226036    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d98bf66b-t4kh5"]
Sep 08 13:19:26.237867 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.237838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8"
Sep 08 13:19:26.248413 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.248390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d98bf66b-t4kh5"]
Sep 08 13:19:26.248519 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.248481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.356888 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.356853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-trusted-ca\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.356888 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.356888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgjc\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-kube-api-access-mhgjc\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.356917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-ca-trust-extracted\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.356985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-bound-sa-token\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.357032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-image-registry-private-configuration\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357145 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.357051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-certificates\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357145 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.357099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-tls\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.357246 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.357158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-installation-pull-secrets\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.424402 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.424327    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:19:26.428313 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:26.428291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4bb584_62d0_494c_b020_a306692b56d0.slice/crio-676ce5234c5ca33978ceff6616714491a21bff1aa5fdae368a147308c4927104 WatchSource:0}: Error finding container 676ce5234c5ca33978ceff6616714491a21bff1aa5fdae368a147308c4927104: Status 404 returned error can't find the container with id 676ce5234c5ca33978ceff6616714491a21bff1aa5fdae368a147308c4927104
Sep 08 13:19:26.457998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.457969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-installation-pull-secrets\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.457998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-trusted-ca\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgjc\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-kube-api-access-mhgjc\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-ca-trust-extracted\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-bound-sa-token\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-image-registry-private-configuration\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-certificates\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458236 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-tls\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.458740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.458679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-ca-trust-extracted\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.459456 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.459420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-certificates\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.459565 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.459496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-trusted-ca\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.460461 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.460442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-installation-pull-secrets\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.460543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.460496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-image-registry-private-configuration\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.461138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.461123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-registry-tls\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.466668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.466643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-bound-sa-token\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.467110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.467095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgjc\" (UniqueName: \"kubernetes.io/projected/95ac8419-8dd3-4b06-bd05-caa2393d1d1a-kube-api-access-mhgjc\") pod \"image-registry-6d98bf66b-t4kh5\" (UID: \"95ac8419-8dd3-4b06-bd05-caa2393d1d1a\") " pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.528492 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.528468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qhdh8"]
Sep 08 13:19:26.531391 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:26.531366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468d9440_90f8_4110_a67b_d6aec9a871ad.slice/crio-e507c662b4d6ebda165d22ab7cebf1abd0b7a681d64ef3fd5ae5be947e3bc06f WatchSource:0}: Error finding container e507c662b4d6ebda165d22ab7cebf1abd0b7a681d64ef3fd5ae5be947e3bc06f: Status 404 returned error can't find the container with id e507c662b4d6ebda165d22ab7cebf1abd0b7a681d64ef3fd5ae5be947e3bc06f
Sep 08 13:19:26.558216 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.558190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:26.761836 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.761791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:26.764006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.763977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96d2294-11ce-403e-9f7f-33d0106bc05e-metrics-tls\") pod \"dns-default-6xdwz\" (UID: \"a96d2294-11ce-403e-9f7f-33d0106bc05e\") " pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:26.852230 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.852198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d98bf66b-t4kh5"]
Sep 08 13:19:26.856766 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:26.856737    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ac8419_8dd3_4b06_bd05_caa2393d1d1a.slice/crio-8014d0dadb831d7dffa631c4618d58b634b22738ec9e690cb8fec151bd4d7f25 WatchSource:0}: Error finding container 8014d0dadb831d7dffa631c4618d58b634b22738ec9e690cb8fec151bd4d7f25: Status 404 returned error can't find the container with id 8014d0dadb831d7dffa631c4618d58b634b22738ec9e690cb8fec151bd4d7f25
Sep 08 13:19:26.893836 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.893814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kwlvq\""
Sep 08 13:19:26.901920 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:26.901895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:27.113407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.113322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5" event={"ID":"95ac8419-8dd3-4b06-bd05-caa2393d1d1a","Type":"ContainerStarted","Data":"155d950bbcd3b8a0e587a821de907449d776c08a7fb62e309fea40997b4bbc8a"}
Sep 08 13:19:27.113587 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.113478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:27.113587 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.113524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5" event={"ID":"95ac8419-8dd3-4b06-bd05-caa2393d1d1a","Type":"ContainerStarted","Data":"8014d0dadb831d7dffa631c4618d58b634b22738ec9e690cb8fec151bd4d7f25"}
Sep 08 13:19:27.114656 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.114630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8" event={"ID":"468d9440-90f8-4110-a67b-d6aec9a871ad","Type":"ContainerStarted","Data":"e507c662b4d6ebda165d22ab7cebf1abd0b7a681d64ef3fd5ae5be947e3bc06f"}
Sep 08 13:19:27.116508 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.116429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" event={"ID":"fa4bb584-62d0-494c-b020-a306692b56d0","Type":"ContainerStarted","Data":"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"}
Sep 08 13:19:27.116508 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.116476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" event={"ID":"fa4bb584-62d0-494c-b020-a306692b56d0","Type":"ContainerStarted","Data":"676ce5234c5ca33978ceff6616714491a21bff1aa5fdae368a147308c4927104"}
Sep 08 13:19:27.116687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.116561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:27.136107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.136055    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5" podStartSLOduration=1.136036198 podStartE2EDuration="1.136036198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:27.135108206 +0000 UTC m=+163.085762159" watchObservedRunningTime="2026-09-08 13:19:27.136036198 +0000 UTC m=+163.086690150"
Sep 08 13:19:27.158760 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.158711    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" podStartSLOduration=34.158693721 podStartE2EDuration="34.158693721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:27.156925816 +0000 UTC m=+163.107579771" watchObservedRunningTime="2026-09-08 13:19:27.158693721 +0000 UTC m=+163.109347675"
Sep 08 13:19:27.196408 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:27.196253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xdwz"]
Sep 08 13:19:27.199191 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:27.199144    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96d2294_11ce_403e_9f7f_33d0106bc05e.slice/crio-77fbc8b11a39d1b4ad035e070c5486f8260cf8478aa0ab807d7c538a14e40941 WatchSource:0}: Error finding container 77fbc8b11a39d1b4ad035e070c5486f8260cf8478aa0ab807d7c538a14e40941: Status 404 returned error can't find the container with id 77fbc8b11a39d1b4ad035e070c5486f8260cf8478aa0ab807d7c538a14e40941
Sep 08 13:19:28.121642 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:28.121551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8" event={"ID":"468d9440-90f8-4110-a67b-d6aec9a871ad","Type":"ContainerStarted","Data":"0292517cd7f93b0f93a3d2872e892505194ebc123bfdcc5db8581f38578bd6ec"}
Sep 08 13:19:28.122809 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:28.122782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xdwz" event={"ID":"a96d2294-11ce-403e-9f7f-33d0106bc05e","Type":"ContainerStarted","Data":"77fbc8b11a39d1b4ad035e070c5486f8260cf8478aa0ab807d7c538a14e40941"}
Sep 08 13:19:28.138913 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:28.138872    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-qhdh8" podStartSLOduration=32.946303767 podStartE2EDuration="34.138858148s" totalImagesPullingTime="1.192554381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:28.137484787 +0000 UTC m=+164.088138739" watchObservedRunningTime="2026-09-08 13:19:28.138858148 +0000 UTC m=+164.089512100"
Sep 08 13:19:29.483681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:29.483643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:29.485979 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:29.485956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ls8l\" (UID: \"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d\") " pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:29.778722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:29.778632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7ls8l"
Sep 08 13:19:30.052434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:30.052392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7ls8l"]
Sep 08 13:19:30.055155 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:30.055128    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf8b852_d325_4d6e_9ff1_e4dc3b9c6c1d.slice/crio-7139e15ccddd525e694fff30cd71f8ae0f73faf22f45f12440c5c9c1f3c05679 WatchSource:0}: Error finding container 7139e15ccddd525e694fff30cd71f8ae0f73faf22f45f12440c5c9c1f3c05679: Status 404 returned error can't find the container with id 7139e15ccddd525e694fff30cd71f8ae0f73faf22f45f12440c5c9c1f3c05679
Sep 08 13:19:30.130456 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:30.130432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ls8l" event={"ID":"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d","Type":"ContainerStarted","Data":"44b34de5cba43bbaf236b616fe69beff33ff2a860e0c2eae1db6ea495a9fc28f"}
Sep 08 13:19:30.130552 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:30.130483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ls8l" event={"ID":"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d","Type":"ContainerStarted","Data":"7139e15ccddd525e694fff30cd71f8ae0f73faf22f45f12440c5c9c1f3c05679"}
Sep 08 13:19:31.136702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.136661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ls8l" event={"ID":"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d","Type":"ContainerStarted","Data":"09604b642811dd93ac51466d3076644bf312d350f320b506a4c625f3483f2d73"}
Sep 08 13:19:31.817607 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.817573    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"]
Sep 08 13:19:31.819676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.819657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:31.822341 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.822307    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 13:19:31.822808 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.822787    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rpzkz\""
Sep 08 13:19:31.830639 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.830596    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"]
Sep 08 13:19:31.905232 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:31.905193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-9sbzj\" (UID: \"3ec78bb5-b335-4874-9a24-37b4af6c260a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:32.006522 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:32.006477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-9sbzj\" (UID: \"3ec78bb5-b335-4874-9a24-37b4af6c260a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:32.006697 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:32.006655    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:32.006756 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:32.006724    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates podName:3ec78bb5-b335-4874-9a24-37b4af6c260a nodeName:}" failed. No retries permitted until 2026-09-08 13:19:32.506707241 +0000 UTC m=+168.457361175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-9sbzj" (UID: "3ec78bb5-b335-4874-9a24-37b4af6c260a") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:32.510068 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:32.510019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-9sbzj\" (UID: \"3ec78bb5-b335-4874-9a24-37b4af6c260a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:32.513143 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:32.513111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3ec78bb5-b335-4874-9a24-37b4af6c260a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-9sbzj\" (UID: \"3ec78bb5-b335-4874-9a24-37b4af6c260a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:32.730940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:32.730916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:32.999013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:32.998981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"]
Sep 08 13:19:33.001793 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:33.001769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec78bb5_b335_4874_9a24_37b4af6c260a.slice/crio-d51954c44adf368f7c7b201922bb498e160e0eecff2bf6ebda9a1fd4ff9395a8 WatchSource:0}: Error finding container d51954c44adf368f7c7b201922bb498e160e0eecff2bf6ebda9a1fd4ff9395a8: Status 404 returned error can't find the container with id d51954c44adf368f7c7b201922bb498e160e0eecff2bf6ebda9a1fd4ff9395a8
Sep 08 13:19:33.148047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.147947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xdwz" event={"ID":"a96d2294-11ce-403e-9f7f-33d0106bc05e","Type":"ContainerStarted","Data":"281a67dda66024a0d467c542d4c535257e2e1f9b04a4ac91fe4fe14afb4e0a83"}
Sep 08 13:19:33.148047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.148011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xdwz" event={"ID":"a96d2294-11ce-403e-9f7f-33d0106bc05e","Type":"ContainerStarted","Data":"4cdc50a3b1531dc93f69480b324fcdbc6c7fcdaeb40f13863dc1386016432453"}
Sep 08 13:19:33.148047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.148040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:33.149209 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.149185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj" event={"ID":"3ec78bb5-b335-4874-9a24-37b4af6c260a","Type":"ContainerStarted","Data":"d51954c44adf368f7c7b201922bb498e160e0eecff2bf6ebda9a1fd4ff9395a8"}
Sep 08 13:19:33.150985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.150964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ls8l" event={"ID":"eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d","Type":"ContainerStarted","Data":"f45c75a923a5cd4381b154723a7f5b56b0ef38c745eb1d4afc264da96e870535"}
Sep 08 13:19:33.166234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.166197    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6xdwz" podStartSLOduration=130.28810609 podStartE2EDuration="2m15.166184467s" totalImagesPullingTime="4.878078377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:33.165975695 +0000 UTC m=+169.116629647" watchObservedRunningTime="2026-09-08 13:19:33.166184467 +0000 UTC m=+169.116838416"
Sep 08 13:19:33.209681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:33.209644    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7ls8l" podStartSLOduration=33.715498437 podStartE2EDuration="36.209630975s" totalImagesPullingTime="2.494132538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:33.209352612 +0000 UTC m=+169.160006566" watchObservedRunningTime="2026-09-08 13:19:33.209630975 +0000 UTC m=+169.160284927"
Sep 08 13:19:34.533151 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:34.533122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:19:35.159003 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:35.158964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj" event={"ID":"3ec78bb5-b335-4874-9a24-37b4af6c260a","Type":"ContainerStarted","Data":"8a47ed1f94e8df3221043046190d8ccf40258bff5caa0bd896bd6f5d84c11303"}
Sep 08 13:19:35.159196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:35.159149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:35.163697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:35.163667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj"
Sep 08 13:19:35.178395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:35.178274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-9sbzj" podStartSLOduration=2.6802468299999997 podStartE2EDuration="4.178258291s" totalImagesPullingTime="1.498011461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:35.177677983 +0000 UTC m=+171.128331935" watchObservedRunningTime="2026-09-08 13:19:35.178258291 +0000 UTC m=+171.128912244"
Sep 08 13:19:41.266615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.266533    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"]
Sep 08 13:19:41.269793 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.269773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.272384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.272356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:19:41.272384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.272381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 13:19:41.272529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.272356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:19:41.273402 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.273385    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9pj8g\""
Sep 08 13:19:41.273488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.273399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:19:41.273488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.273418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:19:41.281145 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.281127    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"]
Sep 08 13:19:41.324290 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.324259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rq24n"]
Sep 08 13:19:41.327368 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.327353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.330294 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.330267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:19:41.330437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.330301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:19:41.330437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.330322    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:19:41.330437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.330320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4t6zk\""
Sep 08 13:19:41.382651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-root\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.382651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-textfile\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.382828 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-metrics-client-ca\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.382828 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxpn\" (UniqueName: \"kubernetes.io/projected/db52b67a-35a7-457c-8b67-d01d9758846a-kube-api-access-kgxpn\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.382828 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-wtmp\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.382828 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.383008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.383008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-sys\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.383008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.383008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.382961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.383008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.383000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rms\" (UniqueName: \"kubernetes.io/projected/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-kube-api-access-h9rms\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.383259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.383054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.383259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.383102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.483912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.483880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.483922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.483940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-sys\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.483956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.483989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rms\" (UniqueName: \"kubernetes.io/projected/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-kube-api-access-h9rms\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-sys\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-root\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-textfile\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-metrics-client-ca\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxpn\" (UniqueName: \"kubernetes.io/projected/db52b67a-35a7-457c-8b67-d01d9758846a-kube-api-access-kgxpn\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:41.484268    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-wtmp\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:41.484354    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls podName:db52b67a-35a7-457c-8b67-d01d9758846a nodeName:}" failed. No retries permitted until 2026-09-08 13:19:41.984331702 +0000 UTC m=+177.934985652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls") pod "node-exporter-rq24n" (UID: "db52b67a-35a7-457c-8b67-d01d9758846a") : secret "node-exporter-tls" not found
Sep 08 13:19:41.484454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-wtmp\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-textfile\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/db52b67a-35a7-457c-8b67-d01d9758846a-root\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.484933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.484933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.484931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db52b67a-35a7-457c-8b67-d01d9758846a-metrics-client-ca\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.486473 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.486454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.486591 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.486574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.486929 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.486908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.494976 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.494945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxpn\" (UniqueName: \"kubernetes.io/projected/db52b67a-35a7-457c-8b67-d01d9758846a-kube-api-access-kgxpn\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.495110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.495095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rms\" (UniqueName: \"kubernetes.io/projected/0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7-kube-api-access-h9rms\") pod \"openshift-state-metrics-657fd6c54-wbxhc\" (UID: \"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.578572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.578503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"
Sep 08 13:19:41.850655 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.850570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc"]
Sep 08 13:19:41.853383 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:41.853354    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0574ff62_e5c2_46e2_b6a0_a2bc2f9941b7.slice/crio-3e9e90c5160f10cf6ba3cb5ca9a1c092a1540af9e16a397976d8eb7b52026e71 WatchSource:0}: Error finding container 3e9e90c5160f10cf6ba3cb5ca9a1c092a1540af9e16a397976d8eb7b52026e71: Status 404 returned error can't find the container with id 3e9e90c5160f10cf6ba3cb5ca9a1c092a1540af9e16a397976d8eb7b52026e71
Sep 08 13:19:41.988562 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.988531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:41.990717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:41.990694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/db52b67a-35a7-457c-8b67-d01d9758846a-node-exporter-tls\") pod \"node-exporter-rq24n\" (UID: \"db52b67a-35a7-457c-8b67-d01d9758846a\") " pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:42.187421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:42.187341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc" event={"ID":"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7","Type":"ContainerStarted","Data":"8f623679a4647a183ecb1f69c95e41c34b86f86d96f35c2a12a32b9a76582f80"}
Sep 08 13:19:42.187421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:42.187401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc" event={"ID":"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7","Type":"ContainerStarted","Data":"5e19b7189f2c8cf7f274f40ba02b4101f8d75a33aef44ff3037c0360a98b5ced"}
Sep 08 13:19:42.187421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:42.187419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc" event={"ID":"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7","Type":"ContainerStarted","Data":"3e9e90c5160f10cf6ba3cb5ca9a1c092a1540af9e16a397976d8eb7b52026e71"}
Sep 08 13:19:42.236280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:42.236249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rq24n"
Sep 08 13:19:42.246443 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:42.246410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52b67a_35a7_457c_8b67_d01d9758846a.slice/crio-913ad9f0e86b648f9627e325394d3f279118bcb64824d4c72a582567bacd6136 WatchSource:0}: Error finding container 913ad9f0e86b648f9627e325394d3f279118bcb64824d4c72a582567bacd6136: Status 404 returned error can't find the container with id 913ad9f0e86b648f9627e325394d3f279118bcb64824d4c72a582567bacd6136
Sep 08 13:19:43.156689 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.156621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6xdwz"
Sep 08 13:19:43.193491 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.193454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc" event={"ID":"0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7","Type":"ContainerStarted","Data":"1aecfb1ce34edcfa4bbe533dcde5e192ecf8868b8c670e8b92251bed6abc110e"}
Sep 08 13:19:43.194893 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.194870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq24n" event={"ID":"db52b67a-35a7-457c-8b67-d01d9758846a","Type":"ContainerStarted","Data":"913ad9f0e86b648f9627e325394d3f279118bcb64824d4c72a582567bacd6136"}
Sep 08 13:19:43.212891 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.212821    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wbxhc" podStartSLOduration=1.347403247 podStartE2EDuration="2.212804652s" totalImagesPullingTime="865.401405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:43.21147875 +0000 UTC m=+179.162132703" watchObservedRunningTime="2026-09-08 13:19:43.212804652 +0000 UTC m=+179.163458830"
Sep 08 13:19:43.274968 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.274939    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8c574fdc9-45xxk"]
Sep 08 13:19:43.278556 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.278530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.282792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.282772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 13:19:43.282921 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.282902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 13:19:43.282974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.282951    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w9sfv\""
Sep 08 13:19:43.283071 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.283053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e8ij2l9s53de7\""
Sep 08 13:19:43.283071 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.283066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 13:19:43.283188 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.283092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 13:19:43.283526 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.283512    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 13:19:43.293408 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.293381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8c574fdc9-45xxk"]
Sep 08 13:19:43.400452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11fd803f-04a2-46e1-bc00-5ae70a190581-metrics-client-ca\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq29k\" (UniqueName: \"kubernetes.io/projected/11fd803f-04a2-46e1-bc00-5ae70a190581-kube-api-access-gq29k\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-grpc-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.400730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.400705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.501815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.501782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.501961 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.501822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.501961 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.501872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.501961 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.501933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11fd803f-04a2-46e1-bc00-5ae70a190581-metrics-client-ca\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.502115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.501967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.502115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.502000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq29k\" (UniqueName: \"kubernetes.io/projected/11fd803f-04a2-46e1-bc00-5ae70a190581-kube-api-access-gq29k\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.502115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.502038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-grpc-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.502115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.502069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.502705 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.502670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11fd803f-04a2-46e1-bc00-5ae70a190581-metrics-client-ca\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.504818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.504792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.504818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.504809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.504941 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.504835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.504941 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.504921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.505097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.505075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-grpc-tls\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.505206 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.505188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/11fd803f-04a2-46e1-bc00-5ae70a190581-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.510561 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.510544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq29k\" (UniqueName: \"kubernetes.io/projected/11fd803f-04a2-46e1-bc00-5ae70a190581-kube-api-access-gq29k\") pod \"thanos-querier-8c574fdc9-45xxk\" (UID: \"11fd803f-04a2-46e1-bc00-5ae70a190581\") " pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.587664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.587629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:43.863195 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:43.863089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8c574fdc9-45xxk"]
Sep 08 13:19:43.866743 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:43.866707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fd803f_04a2_46e1_bc00_5ae70a190581.slice/crio-d5e8da2b5bf560709f9f93789ed245699bf3aecf5b5a969a991c8f3c86c9d1d2 WatchSource:0}: Error finding container d5e8da2b5bf560709f9f93789ed245699bf3aecf5b5a969a991c8f3c86c9d1d2: Status 404 returned error can't find the container with id d5e8da2b5bf560709f9f93789ed245699bf3aecf5b5a969a991c8f3c86c9d1d2
Sep 08 13:19:44.200667 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:44.200589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"d5e8da2b5bf560709f9f93789ed245699bf3aecf5b5a969a991c8f3c86c9d1d2"}
Sep 08 13:19:44.202322 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:44.202293    2580 generic.go:356] "Generic (PLEG): container finished" podID="db52b67a-35a7-457c-8b67-d01d9758846a" containerID="5db0a882658fe3c1723b3549c3a090b9e0316d6d1c326bef9e2383dfb9a03880" exitCode=0
Sep 08 13:19:44.202432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:44.202378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq24n" event={"ID":"db52b67a-35a7-457c-8b67-d01d9758846a","Type":"ContainerDied","Data":"5db0a882658fe3c1723b3549c3a090b9e0316d6d1c326bef9e2383dfb9a03880"}
Sep 08 13:19:45.207754 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.207718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq24n" event={"ID":"db52b67a-35a7-457c-8b67-d01d9758846a","Type":"ContainerStarted","Data":"fa7a831eba270c88b5c6590f7bdb02fd65201064f196922c2a7f874d58ddbb11"}
Sep 08 13:19:45.208109 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.207776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rq24n" event={"ID":"db52b67a-35a7-457c-8b67-d01d9758846a","Type":"ContainerStarted","Data":"a48ecbe2fba72e13dd510f6ac05e16ea76a41757ef5a95b10ba8a3a6581d0f6d"}
Sep 08 13:19:45.231139 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.231080    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rq24n" podStartSLOduration=3.227534986 podStartE2EDuration="4.231057518s" totalImagesPullingTime="938.842893ms" totalInitContainerRuntime="64.679639ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:45.228586435 +0000 UTC m=+181.179240387" watchObservedRunningTime="2026-09-08 13:19:45.231057518 +0000 UTC m=+181.181711472"
Sep 08 13:19:45.802555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.802530    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65789ddc66-f9zf2"]
Sep 08 13:19:45.805932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.805915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.808437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.808417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 13:19:45.809639 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.809619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-p2q6x\""
Sep 08 13:19:45.809758 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.809741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 13:19:45.809825 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.809741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-i53qofdphfkv\""
Sep 08 13:19:45.809825 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.809786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 13:19:45.809926 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.809857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 13:19:45.818229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.818209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65789ddc66-f9zf2"]
Sep 08 13:19:45.923724 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bfa675da-98e3-40bc-829a-f4c166fd1b62-audit-log\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.923818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.923818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-tls\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.923922 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-client-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.923974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxqf\" (UniqueName: \"kubernetes.io/projected/bfa675da-98e3-40bc-829a-f4c166fd1b62-kube-api-access-jrxqf\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.924022 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.923982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-client-certs\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:45.924078 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:45.924044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-metrics-server-audit-profiles\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025269 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-client-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxqf\" (UniqueName: \"kubernetes.io/projected/bfa675da-98e3-40bc-829a-f4c166fd1b62-kube-api-access-jrxqf\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-client-certs\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-metrics-server-audit-profiles\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025558 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bfa675da-98e3-40bc-829a-f4c166fd1b62-audit-log\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025653 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-tls\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.025818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.025785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bfa675da-98e3-40bc-829a-f4c166fd1b62-audit-log\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.026199 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.026160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.026334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.026316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bfa675da-98e3-40bc-829a-f4c166fd1b62-metrics-server-audit-profiles\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.027777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.027757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-client-certs\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.027869 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.027793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-client-ca-bundle\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.027869 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.027797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bfa675da-98e3-40bc-829a-f4c166fd1b62-secret-metrics-server-tls\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.036092 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.036074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxqf\" (UniqueName: \"kubernetes.io/projected/bfa675da-98e3-40bc-829a-f4c166fd1b62-kube-api-access-jrxqf\") pod \"metrics-server-65789ddc66-f9zf2\" (UID: \"bfa675da-98e3-40bc-829a-f4c166fd1b62\") " pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.062481 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.062458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"]
Sep 08 13:19:46.066877 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.066862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.069505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.069487    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fktcl\""
Sep 08 13:19:46.069579 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.069522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 13:19:46.076063 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.076041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"]
Sep 08 13:19:46.115435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.115412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:19:46.126584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.126562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll7v\" (UniqueName: \"kubernetes.io/projected/d8f7fe01-c204-4e40-8afd-deb24e8916c0-kube-api-access-7ll7v\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.126676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.126627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8f7fe01-c204-4e40-8afd-deb24e8916c0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.217469 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.217435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"d168642405d6592e1295fe6a80782252659480937f99bb96140ae3f8fece2c76"}
Sep 08 13:19:46.217830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.217520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"e5ecc1c4c57f421e36b6e98c5e4f41283910481f3cf9f0edfab3d7734dcd7132"}
Sep 08 13:19:46.217830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.217546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"c61a28a4128d43799b1232a0cf7f14afb76d1a79635bbe91b44d8e18a5f5556e"}
Sep 08 13:19:46.226998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.226972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8f7fe01-c204-4e40-8afd-deb24e8916c0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.227062 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.227034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll7v\" (UniqueName: \"kubernetes.io/projected/d8f7fe01-c204-4e40-8afd-deb24e8916c0-kube-api-access-7ll7v\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.229295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.229271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8f7fe01-c204-4e40-8afd-deb24e8916c0-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.235918 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.235897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll7v\" (UniqueName: \"kubernetes.io/projected/d8f7fe01-c204-4e40-8afd-deb24e8916c0-kube-api-access-7ll7v\") pod \"monitoring-plugin-85bbbbbf4-zrcm7\" (UID: \"d8f7fe01-c204-4e40-8afd-deb24e8916c0\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.376529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.376467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:46.386511 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.386486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65789ddc66-f9zf2"]
Sep 08 13:19:46.389121 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:46.389091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa675da_98e3_40bc_829a_f4c166fd1b62.slice/crio-2536e97a74d05f74e5e806cbf922aca60f0766af8adfb7a051787bbcadd88142 WatchSource:0}: Error finding container 2536e97a74d05f74e5e806cbf922aca60f0766af8adfb7a051787bbcadd88142: Status 404 returned error can't find the container with id 2536e97a74d05f74e5e806cbf922aca60f0766af8adfb7a051787bbcadd88142
Sep 08 13:19:46.562567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.562534    2580 patch_prober.go:28] interesting pod/image-registry-6d98bf66b-t4kh5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:19:46.562707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.562587    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5" podUID="95ac8419-8dd3-4b06-bd05-caa2393d1d1a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:19:46.645697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:46.645616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"]
Sep 08 13:19:46.648574 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:46.648547    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f7fe01_c204_4e40_8afd_deb24e8916c0.slice/crio-f08eef7083772a69d48b0bcf14ff30467f3b8fa33ff23256edd1704a46b265be WatchSource:0}: Error finding container f08eef7083772a69d48b0bcf14ff30467f3b8fa33ff23256edd1704a46b265be: Status 404 returned error can't find the container with id f08eef7083772a69d48b0bcf14ff30467f3b8fa33ff23256edd1704a46b265be
Sep 08 13:19:47.123620 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:47.123584    2580 patch_prober.go:28] interesting pod/image-registry-5db4cddb66-4cjpn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:19:47.123783 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:47.123653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:19:47.223748 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:47.223700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2" event={"ID":"bfa675da-98e3-40bc-829a-f4c166fd1b62","Type":"ContainerStarted","Data":"2536e97a74d05f74e5e806cbf922aca60f0766af8adfb7a051787bbcadd88142"}
Sep 08 13:19:47.225376 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:47.225344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7" event={"ID":"d8f7fe01-c204-4e40-8afd-deb24e8916c0","Type":"ContainerStarted","Data":"f08eef7083772a69d48b0bcf14ff30467f3b8fa33ff23256edd1704a46b265be"}
Sep 08 13:19:48.128158 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.128124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d98bf66b-t4kh5"
Sep 08 13:19:48.233397 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.233334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2" event={"ID":"bfa675da-98e3-40bc-829a-f4c166fd1b62","Type":"ContainerStarted","Data":"62bfe16a43326a16040757ca7c565f659b1e7579af46dcd0efc6402507f75eef"}
Sep 08 13:19:48.237144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.237117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"16e38a28a35c356b3b12517a36571c7a125e98754023ede7c69e336235fa0917"}
Sep 08 13:19:48.237301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.237199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"f66bc571107667ca8ad598d0103bba698bbe702e166365f395ec3042ae22d610"}
Sep 08 13:19:48.237301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.237244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" event={"ID":"11fd803f-04a2-46e1-bc00-5ae70a190581","Type":"ContainerStarted","Data":"4adf72725c1f472a645376a1dcb6f370bf7652dbcb00dc196160a9c1a5527f94"}
Sep 08 13:19:48.237301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.237290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:48.254390 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.254341    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2" podStartSLOduration=1.764331139 podStartE2EDuration="3.254321059s" totalImagesPullingTime="1.48998992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:48.251989305 +0000 UTC m=+184.202643260" watchObservedRunningTime="2026-09-08 13:19:48.254321059 +0000 UTC m=+184.204975011"
Sep 08 13:19:48.277937 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:48.277864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk" podStartSLOduration=2.047045293 podStartE2EDuration="5.277848428s" totalImagesPullingTime="3.230803135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:48.276418051 +0000 UTC m=+184.227072218" watchObservedRunningTime="2026-09-08 13:19:48.277848428 +0000 UTC m=+184.228502379"
Sep 08 13:19:49.067257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.067179    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:19:49.070554 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.070536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.073428 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.073402    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 13:19:49.073553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.073434    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 13:19:49.073553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.073461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n2fsh\""
Sep 08 13:19:49.073553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.073470    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 13:19:49.073553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.073402    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 13:19:49.074357 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.074339    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 13:19:49.074357 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.074352    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 13:19:49.074498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.074449    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 13:19:49.083533 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.083507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:19:49.155086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.155284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.155284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvdv\" (UniqueName: \"kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.155284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.155443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.155443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.155361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.242505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.242464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7" event={"ID":"d8f7fe01-c204-4e40-8afd-deb24e8916c0","Type":"ContainerStarted","Data":"0dec29ff5f55d56fe58f2b6b85e07b7b7028c4efa8d07c2fdd61e32330b8199e"}
Sep 08 13:19:49.243000 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.242982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:49.247482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.247461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7"
Sep 08 13:19:49.256400 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.256448 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.256448 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvdv\" (UniqueName: \"kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.256524 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.256560 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.256597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.256563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.257180 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.257146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.257269 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.257230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.257269 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.257230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.258891 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.258868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.258988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.258954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.261320 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.261280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-zrcm7" podStartSLOduration=1.105084537 podStartE2EDuration="3.261267086s" totalImagesPullingTime="2.156182549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:49.259874925 +0000 UTC m=+185.210528878" watchObservedRunningTime="2026-09-08 13:19:49.261267086 +0000 UTC m=+185.211921095"
Sep 08 13:19:49.269422 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.269401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvdv\" (UniqueName: \"kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv\") pod \"console-7977967d58-nxj6r\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") " pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.380379 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.380299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:49.656812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:49.656730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:19:49.661257 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:49.661233    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a003052_141f_47b5_a458_38abf0b6c058.slice/crio-fcf879caa4217e81a8bf6008b3ca3646ddd6d7a9502af32663f00ff097edf1f9 WatchSource:0}: Error finding container fcf879caa4217e81a8bf6008b3ca3646ddd6d7a9502af32663f00ff097edf1f9: Status 404 returned error can't find the container with id fcf879caa4217e81a8bf6008b3ca3646ddd6d7a9502af32663f00ff097edf1f9
Sep 08 13:19:50.247514 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:50.247478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7977967d58-nxj6r" event={"ID":"9a003052-141f-47b5-a458-38abf0b6c058","Type":"ContainerStarted","Data":"fcf879caa4217e81a8bf6008b3ca3646ddd6d7a9502af32663f00ff097edf1f9"}
Sep 08 13:19:52.139525 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.139486    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" containerName="registry" containerID="cri-o://af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89" gracePeriod=30
Sep 08 13:19:52.878842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.878819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:52.990708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr762\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.990900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.991225 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.990919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration\") pod \"fa4bb584-62d0-494c-b020-a306692b56d0\" (UID: \"fa4bb584-62d0-494c-b020-a306692b56d0\") "
Sep 08 13:19:52.991283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.991245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:52.991377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.991348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:52.993297 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.993269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:52.993297 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.993282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:52.993458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.993383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762" (OuterVolumeSpecName: "kube-api-access-jr762") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "kube-api-access-jr762". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:52.993458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.993431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:52.993610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.993583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:52.999256 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:52.999236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fa4bb584-62d0-494c-b020-a306692b56d0" (UID: "fa4bb584-62d0-494c-b020-a306692b56d0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092213    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-trusted-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092259    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr762\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-kube-api-access-jr762\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092277    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-registry-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092293    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa4bb584-62d0-494c-b020-a306692b56d0-bound-sa-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092313    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa4bb584-62d0-494c-b020-a306692b56d0-registry-certificates\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092327    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-installation-pull-secrets\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092341    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa4bb584-62d0-494c-b020-a306692b56d0-ca-trust-extracted\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.096263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.092356    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa4bb584-62d0-494c-b020-a306692b56d0-image-registry-private-configuration\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:19:53.257261 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.257184    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:19:53.257608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.257511    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" containerName="registry"
Sep 08 13:19:53.257608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.257523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" containerName="registry"
Sep 08 13:19:53.257608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.257575    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" containerName="registry"
Sep 08 13:19:53.262435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.262413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.263513 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.263484    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4bb584-62d0-494c-b020-a306692b56d0" containerID="af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89" exitCode=0
Sep 08 13:19:53.263611 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.263541    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn"
Sep 08 13:19:53.263611 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.263572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" event={"ID":"fa4bb584-62d0-494c-b020-a306692b56d0","Type":"ContainerDied","Data":"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"}
Sep 08 13:19:53.263723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.263615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db4cddb66-4cjpn" event={"ID":"fa4bb584-62d0-494c-b020-a306692b56d0","Type":"ContainerDied","Data":"676ce5234c5ca33978ceff6616714491a21bff1aa5fdae368a147308c4927104"}
Sep 08 13:19:53.263723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.263637    2580 scope.go:117] "RemoveContainer" containerID="af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"
Sep 08 13:19:53.265432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.265412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7977967d58-nxj6r" event={"ID":"9a003052-141f-47b5-a458-38abf0b6c058","Type":"ContainerStarted","Data":"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"}
Sep 08 13:19:53.271038 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.270751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 13:19:53.274201 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.274141    2580 scope.go:117] "RemoveContainer" containerID="af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"
Sep 08 13:19:53.274684 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:19:53.274571    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89\": container with ID starting with af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89 not found: ID does not exist" containerID="af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"
Sep 08 13:19:53.274684 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.274605    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89"} err="failed to get container status \"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89\": rpc error: code = NotFound desc = could not find container \"af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89\": container with ID starting with af2fa6d70036337e4cc758107fafb858a937900c584ec10ecf6e9bb6e52b0c89 not found: ID does not exist"
Sep 08 13:19:53.275748 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.275727    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:19:53.306897 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.306870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:19:53.310041 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.310019    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5db4cddb66-4cjpn"]
Sep 08 13:19:53.332297 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.332255    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7977967d58-nxj6r" podStartSLOduration=1.201549905 podStartE2EDuration="4.332241342s" totalImagesPullingTime="3.130691437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:53.331265827 +0000 UTC m=+189.281919782" watchObservedRunningTime="2026-09-08 13:19:53.332241342 +0000 UTC m=+189.282895293"
Sep 08 13:19:53.394851 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.394812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bl8\" (UniqueName: \"kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.394988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.394892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.395045 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.395035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.395089 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.395075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.395131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.395111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.395210 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.395182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.395262 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.395222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495659 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.495816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.496042 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.495852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bl8\" (UniqueName: \"kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.496470 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.496442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.496576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.496513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.496576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.496543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.496681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.496604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.498202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.498178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.498290 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.498183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.504119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.504097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bl8\" (UniqueName: \"kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8\") pod \"console-56fc448cfb-z8c54\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") " pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.574148 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.574092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:19:53.846999 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:53.846927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:19:53.850007 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:19:53.849979    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe566b2c_6a1e_4466_9572_d622055145ef.slice/crio-61d0c9ba13b1b7a22d1422065be3b6d3a4d86c6cfc3a36862b9b58399414bec4 WatchSource:0}: Error finding container 61d0c9ba13b1b7a22d1422065be3b6d3a4d86c6cfc3a36862b9b58399414bec4: Status 404 returned error can't find the container with id 61d0c9ba13b1b7a22d1422065be3b6d3a4d86c6cfc3a36862b9b58399414bec4
Sep 08 13:19:54.249282 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:54.249255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8c574fdc9-45xxk"
Sep 08 13:19:54.272020 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:54.271986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fc448cfb-z8c54" event={"ID":"be566b2c-6a1e-4466-9572-d622055145ef","Type":"ContainerStarted","Data":"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"}
Sep 08 13:19:54.272452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:54.272037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fc448cfb-z8c54" event={"ID":"be566b2c-6a1e-4466-9572-d622055145ef","Type":"ContainerStarted","Data":"61d0c9ba13b1b7a22d1422065be3b6d3a4d86c6cfc3a36862b9b58399414bec4"}
Sep 08 13:19:54.295557 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:54.295511    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56fc448cfb-z8c54" podStartSLOduration=1.295497433 podStartE2EDuration="1.295497433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:54.293930794 +0000 UTC m=+190.244584746" watchObservedRunningTime="2026-09-08 13:19:54.295497433 +0000 UTC m=+190.246151385"
Sep 08 13:19:54.535227 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:54.535115    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4bb584-62d0-494c-b020-a306692b56d0" path="/var/lib/kubelet/pods/fa4bb584-62d0-494c-b020-a306692b56d0/volumes"
Sep 08 13:19:59.380647 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:59.380601    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:59.381272 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:59.381051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:19:59.387933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:19:59.387908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:20:00.298870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:00.298840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:20:02.304222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:02.304129    2580 generic.go:356] "Generic (PLEG): container finished" podID="ea5563ed-b472-4c08-8ecc-f61b5cd573ee" containerID="a0324ac8c4ce200f5bbc9ebfd02bf0f60bd2a9e0157e03e81c05308799ceb608" exitCode=0
Sep 08 13:20:02.304222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:02.304208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr" event={"ID":"ea5563ed-b472-4c08-8ecc-f61b5cd573ee","Type":"ContainerDied","Data":"a0324ac8c4ce200f5bbc9ebfd02bf0f60bd2a9e0157e03e81c05308799ceb608"}
Sep 08 13:20:02.304709 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:02.304618    2580 scope.go:117] "RemoveContainer" containerID="a0324ac8c4ce200f5bbc9ebfd02bf0f60bd2a9e0157e03e81c05308799ceb608"
Sep 08 13:20:03.310716 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:03.310681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-qnzzr" event={"ID":"ea5563ed-b472-4c08-8ecc-f61b5cd573ee","Type":"ContainerStarted","Data":"c01c3ce3503d731d34e63f2c946776c81a9aa0f36e6886c989a1c32f285f7172"}
Sep 08 13:20:03.574465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:03.574367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:20:03.574465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:03.574458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:20:03.579099 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:03.579077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:20:04.318337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:04.318304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:20:04.373979 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:04.373953    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:20:06.115803 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:06.115763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:20:06.116218 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:06.115843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:20:21.386299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:21.386266    2580 generic.go:356] "Generic (PLEG): container finished" podID="428db53a-2e22-4190-ab14-83725d171d16" containerID="b6019556513ca69217c0ac9b9c8bfb11ce0084724ab1d4e85bd7679ca0211a05" exitCode=0
Sep 08 13:20:21.386615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:21.386296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9" event={"ID":"428db53a-2e22-4190-ab14-83725d171d16","Type":"ContainerDied","Data":"b6019556513ca69217c0ac9b9c8bfb11ce0084724ab1d4e85bd7679ca0211a05"}
Sep 08 13:20:21.386615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:21.386598    2580 scope.go:117] "RemoveContainer" containerID="b6019556513ca69217c0ac9b9c8bfb11ce0084724ab1d4e85bd7679ca0211a05"
Sep 08 13:20:22.392389 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:22.392357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-29zz9" event={"ID":"428db53a-2e22-4190-ab14-83725d171d16","Type":"ContainerStarted","Data":"77b18c14cfcfead8174377d69fbbd336700a3eca209bb9a7ecf90e269d80b55b"}
Sep 08 13:20:26.121873 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:26.121843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:20:26.125620 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:26.125594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65789ddc66-f9zf2"
Sep 08 13:20:28.335921 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:28.335893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xdwz_a96d2294-11ce-403e-9f7f-33d0106bc05e/dns/0.log"
Sep 08 13:20:28.343116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:28.343098    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xdwz_a96d2294-11ce-403e-9f7f-33d0106bc05e/kube-rbac-proxy/0.log"
Sep 08 13:20:28.410263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:28.410242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7sl66_272ac4cc-2f4d-45de-b56b-e83b1918116c/dns-node-resolver/0.log"
Sep 08 13:20:29.393918 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.393880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7977967d58-nxj6r" podUID="9a003052-141f-47b5-a458-38abf0b6c058" containerName="console" containerID="cri-o://f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9" gracePeriod=15
Sep 08 13:20:29.635708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.635681    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:20:29.685421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kvdv\" (UniqueName: \"kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert\") pod \"9a003052-141f-47b5-a458-38abf0b6c058\" (UID: \"9a003052-141f-47b5-a458-38abf0b6c058\") "
Sep 08 13:20:29.685902 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config" (OuterVolumeSpecName: "console-config") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:29.685962 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:29.685962 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.685917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:29.687799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.687780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:29.687863 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.687835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:29.687897 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.687862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv" (OuterVolumeSpecName: "kube-api-access-7kvdv") pod "9a003052-141f-47b5-a458-38abf0b6c058" (UID: "9a003052-141f-47b5-a458-38abf0b6c058"). InnerVolumeSpecName "kube-api-access-7kvdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:29.786444 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786414    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-console-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:29.786444 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786439    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-oauth-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:29.786444 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786450    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-oauth-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:29.786717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786460    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvdv\" (UniqueName: \"kubernetes.io/projected/9a003052-141f-47b5-a458-38abf0b6c058-kube-api-access-7kvdv\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:29.786717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786469    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a003052-141f-47b5-a458-38abf0b6c058-console-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:29.786717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:29.786477    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a003052-141f-47b5-a458-38abf0b6c058-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:20:30.424872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.424830    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a003052-141f-47b5-a458-38abf0b6c058" containerID="f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9" exitCode=0
Sep 08 13:20:30.425505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.424887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7977967d58-nxj6r"
Sep 08 13:20:30.425505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.424917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7977967d58-nxj6r" event={"ID":"9a003052-141f-47b5-a458-38abf0b6c058","Type":"ContainerDied","Data":"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"}
Sep 08 13:20:30.425505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.424994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7977967d58-nxj6r" event={"ID":"9a003052-141f-47b5-a458-38abf0b6c058","Type":"ContainerDied","Data":"fcf879caa4217e81a8bf6008b3ca3646ddd6d7a9502af32663f00ff097edf1f9"}
Sep 08 13:20:30.425505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.425017    2580 scope.go:117] "RemoveContainer" containerID="f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"
Sep 08 13:20:30.434274 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.434243    2580 scope.go:117] "RemoveContainer" containerID="f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"
Sep 08 13:20:30.434542 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:20:30.434499    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9\": container with ID starting with f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9 not found: ID does not exist" containerID="f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"
Sep 08 13:20:30.434581 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.434550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9"} err="failed to get container status \"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9\": rpc error: code = NotFound desc = could not find container \"f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9\": container with ID starting with f938df5ae28dd6db57b61d22b723b5373e48be42158664bf0f65fe3a8d9e85b9 not found: ID does not exist"
Sep 08 13:20:30.447566 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.447545    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:20:30.451612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.451592    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7977967d58-nxj6r"]
Sep 08 13:20:30.535007 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:30.534979    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a003052-141f-47b5-a458-38abf0b6c058" path="/var/lib/kubelet/pods/9a003052-141f-47b5-a458-38abf0b6c058/volumes"
Sep 08 13:20:56.313688 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:56.313642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:20:56.315974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:56.315953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fa069d-7b89-4ce6-823f-613bfb31539a-metrics-certs\") pod \"network-metrics-daemon-4q54b\" (UID: \"05fa069d-7b89-4ce6-823f-613bfb31539a\") " pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:20:56.436249 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:56.436221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxg9n\""
Sep 08 13:20:56.443861 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:56.443841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q54b"
Sep 08 13:20:56.715583 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:56.715560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q54b"]
Sep 08 13:20:56.717820 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:20:56.717796    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fa069d_7b89_4ce6_823f_613bfb31539a.slice/crio-ff6b61eecfac1b3701632b07869750f4db33d2d74c535a14f2b198c8f7060578 WatchSource:0}: Error finding container ff6b61eecfac1b3701632b07869750f4db33d2d74c535a14f2b198c8f7060578: Status 404 returned error can't find the container with id ff6b61eecfac1b3701632b07869750f4db33d2d74c535a14f2b198c8f7060578
Sep 08 13:20:57.533447 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:57.533415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q54b" event={"ID":"05fa069d-7b89-4ce6-823f-613bfb31539a","Type":"ContainerStarted","Data":"ff6b61eecfac1b3701632b07869750f4db33d2d74c535a14f2b198c8f7060578"}
Sep 08 13:20:58.538493 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:58.538450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q54b" event={"ID":"05fa069d-7b89-4ce6-823f-613bfb31539a","Type":"ContainerStarted","Data":"7385f38e6416a33f582c968c4dd50cf1f7ed1b94a88c0e97210e4e3110d2c3a9"}
Sep 08 13:20:58.538902 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:58.538509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q54b" event={"ID":"05fa069d-7b89-4ce6-823f-613bfb31539a","Type":"ContainerStarted","Data":"ee583ebacb0a1a017eeaf6cf6c0a68e8f7729d36bdeb4bba59f39ef6090b9869"}
Sep 08 13:20:58.556886 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:20:58.556839    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4q54b" podStartSLOduration=252.256396813 podStartE2EDuration="4m13.556823954s" totalImagesPullingTime="1.300427141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:58.555738195 +0000 UTC m=+254.506392150" watchObservedRunningTime="2026-09-08 13:20:58.556823954 +0000 UTC m=+254.507477906"
Sep 08 13:21:08.859280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.859204    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:21:08.859742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.859520    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a003052-141f-47b5-a458-38abf0b6c058" containerName="console"
Sep 08 13:21:08.859742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.859531    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a003052-141f-47b5-a458-38abf0b6c058" containerName="console"
Sep 08 13:21:08.859742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.859601    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a003052-141f-47b5-a458-38abf0b6c058" containerName="console"
Sep 08 13:21:08.863567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.863549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.875952 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.875899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:21:08.914211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914533 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck299\" (UniqueName: \"kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:08.914533 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:08.914448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015364 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015364 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck299\" (UniqueName: \"kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015768 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.015768 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.015672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.016091 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.016066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.016232 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.016211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.016353 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.016335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.016606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.016581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.018043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.018021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.018122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.018044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.026215 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.026157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck299\" (UniqueName: \"kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299\") pod \"console-5ffc67fd89-khspk\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") " pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.173087 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.173005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:09.449685 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.449616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:21:09.452794 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:21:09.452767    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ec51bf_c382_404f_b380_3673166f0368.slice/crio-3041c360ad9f30d9fea6955e7b241accb2ac30d3b8fa959abd3a7cbd11202bc4 WatchSource:0}: Error finding container 3041c360ad9f30d9fea6955e7b241accb2ac30d3b8fa959abd3a7cbd11202bc4: Status 404 returned error can't find the container with id 3041c360ad9f30d9fea6955e7b241accb2ac30d3b8fa959abd3a7cbd11202bc4
Sep 08 13:21:09.581975 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.581936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffc67fd89-khspk" event={"ID":"e9ec51bf-c382-404f-b380-3673166f0368","Type":"ContainerStarted","Data":"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"}
Sep 08 13:21:09.582131 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:09.581996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffc67fd89-khspk" event={"ID":"e9ec51bf-c382-404f-b380-3673166f0368","Type":"ContainerStarted","Data":"3041c360ad9f30d9fea6955e7b241accb2ac30d3b8fa959abd3a7cbd11202bc4"}
Sep 08 13:21:19.174069 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.174029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:19.174506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.174109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:19.178779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.178754    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:19.216089 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.216036    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ffc67fd89-khspk" podStartSLOduration=11.216019996 podStartE2EDuration="11.216019996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:09.605721545 +0000 UTC m=+265.556375499" watchObservedRunningTime="2026-09-08 13:21:19.216019996 +0000 UTC m=+275.166673948"
Sep 08 13:21:19.624567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.624536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:21:19.722887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:19.722856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:21:44.424784 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:44.424755    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:21:44.427724 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:44.427699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:21:44.428041 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:44.428023    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:21:44.742084 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:44.742028    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56fc448cfb-z8c54" podUID="be566b2c-6a1e-4466-9572-d622055145ef" containerName="console" containerID="cri-o://9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc" gracePeriod=15
Sep 08 13:21:44.979055 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:44.979023    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:21:45.103677 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103677 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103677 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87bl8\" (UniqueName: \"kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.103919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.103860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle\") pod \"be566b2c-6a1e-4466-9572-d622055145ef\" (UID: \"be566b2c-6a1e-4466-9572-d622055145ef\") "
Sep 08 13:21:45.104121 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.104104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:45.104372 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.104280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:45.104498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.104388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config" (OuterVolumeSpecName: "console-config") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:45.104584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.104531    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:45.105950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.105928    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:45.106052 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.105947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:45.106052 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.105975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8" (OuterVolumeSpecName: "kube-api-access-87bl8") pod "be566b2c-6a1e-4466-9572-d622055145ef" (UID: "be566b2c-6a1e-4466-9572-d622055145ef"). InnerVolumeSpecName "kube-api-access-87bl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:45.204729 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204696    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204729 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204723    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204729 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204734    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bl8\" (UniqueName: \"kubernetes.io/projected/be566b2c-6a1e-4466-9572-d622055145ef-kube-api-access-87bl8\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204742    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be566b2c-6a1e-4466-9572-d622055145ef-console-oauth-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204751    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-oauth-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204759    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-console-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.204940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.204768    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be566b2c-6a1e-4466-9572-d622055145ef-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:21:45.724307 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.724264    2580 generic.go:356] "Generic (PLEG): container finished" podID="be566b2c-6a1e-4466-9572-d622055145ef" containerID="9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc" exitCode=0
Sep 08 13:21:45.724769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.724336    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fc448cfb-z8c54"
Sep 08 13:21:45.724769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.724337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fc448cfb-z8c54" event={"ID":"be566b2c-6a1e-4466-9572-d622055145ef","Type":"ContainerDied","Data":"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"}
Sep 08 13:21:45.724769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.724460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fc448cfb-z8c54" event={"ID":"be566b2c-6a1e-4466-9572-d622055145ef","Type":"ContainerDied","Data":"61d0c9ba13b1b7a22d1422065be3b6d3a4d86c6cfc3a36862b9b58399414bec4"}
Sep 08 13:21:45.724769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.724480    2580 scope.go:117] "RemoveContainer" containerID="9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"
Sep 08 13:21:45.732792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.732773    2580 scope.go:117] "RemoveContainer" containerID="9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"
Sep 08 13:21:45.733058 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:21:45.733038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc\": container with ID starting with 9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc not found: ID does not exist" containerID="9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"
Sep 08 13:21:45.733122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.733071    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc"} err="failed to get container status \"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc\": rpc error: code = NotFound desc = could not find container \"9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc\": container with ID starting with 9ab3eda0f7550c1aec5e55e1849db8401fa8e71803fa2f8686fd6158249049cc not found: ID does not exist"
Sep 08 13:21:45.746101 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.746075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:21:45.750150 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:45.750127    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56fc448cfb-z8c54"]
Sep 08 13:21:46.535229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:21:46.535194    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be566b2c-6a1e-4466-9572-d622055145ef" path="/var/lib/kubelet/pods/be566b2c-6a1e-4466-9572-d622055145ef/volumes"
Sep 08 13:22:30.001665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.001632    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:22:30.002102 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.001968    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be566b2c-6a1e-4466-9572-d622055145ef" containerName="console"
Sep 08 13:22:30.002102 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.001981    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be566b2c-6a1e-4466-9572-d622055145ef" containerName="console"
Sep 08 13:22:30.002102 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.002032    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be566b2c-6a1e-4466-9572-d622055145ef" containerName="console"
Sep 08 13:22:30.004994 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.004977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.020216 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.020189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:22:30.049384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049515 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl99d\" (UniqueName: \"kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049515 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049515 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.049630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.049562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl99d\" (UniqueName: \"kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.150676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.150648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.151320 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.151293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.151441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.151324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.151441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.151341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.151733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.151706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.152964 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.152942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.153179 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.153146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.158968 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.158945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl99d\" (UniqueName: \"kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d\") pod \"console-68d5d99b7b-8w2qk\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") " pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.315255 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.315140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:30.587793 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.587769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:22:30.590399 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:22:30.590373    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a571f7_e83a_4cd8_91f3_bf8576d85a02.slice/crio-dc52cbe95be661e8585fc8f7a2a69675842f0bfd7f6cbc670889ee266f23da48 WatchSource:0}: Error finding container dc52cbe95be661e8585fc8f7a2a69675842f0bfd7f6cbc670889ee266f23da48: Status 404 returned error can't find the container with id dc52cbe95be661e8585fc8f7a2a69675842f0bfd7f6cbc670889ee266f23da48
Sep 08 13:22:30.899309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.899224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d5d99b7b-8w2qk" event={"ID":"19a571f7-e83a-4cd8-91f3-bf8576d85a02","Type":"ContainerStarted","Data":"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"}
Sep 08 13:22:30.899309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.899277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d5d99b7b-8w2qk" event={"ID":"19a571f7-e83a-4cd8-91f3-bf8576d85a02","Type":"ContainerStarted","Data":"dc52cbe95be661e8585fc8f7a2a69675842f0bfd7f6cbc670889ee266f23da48"}
Sep 08 13:22:30.919855 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:30.919804    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68d5d99b7b-8w2qk" podStartSLOduration=1.919788405 podStartE2EDuration="1.919788405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:22:30.918384773 +0000 UTC m=+346.869038722" watchObservedRunningTime="2026-09-08 13:22:30.919788405 +0000 UTC m=+346.870442357"
Sep 08 13:22:40.315663 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:40.315582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:40.316114 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:40.315685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:40.320223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:40.320201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:40.941586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:40.941554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:22:40.992812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:22:40.992781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:23:06.012426 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.012366    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ffc67fd89-khspk" podUID="e9ec51bf-c382-404f-b380-3673166f0368" containerName="console" containerID="cri-o://bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446" gracePeriod=15
Sep 08 13:23:06.258626 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.258601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:23:06.356371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356297    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck299\" (UniqueName: \"kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca\") pod \"e9ec51bf-c382-404f-b380-3673166f0368\" (UID: \"e9ec51bf-c382-404f-b380-3673166f0368\") "
Sep 08 13:23:06.356839 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:23:06.356884 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config" (OuterVolumeSpecName: "console-config") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:23:06.356930 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356921    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-console-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.356972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356937    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-oauth-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.356972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:23:06.356972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.356956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca" (OuterVolumeSpecName: "service-ca") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:23:06.358575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.358550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:23:06.358575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.358569    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:23:06.358759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.358622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299" (OuterVolumeSpecName: "kube-api-access-ck299") pod "e9ec51bf-c382-404f-b380-3673166f0368" (UID: "e9ec51bf-c382-404f-b380-3673166f0368"). InnerVolumeSpecName "kube-api-access-ck299". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:23:06.457482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.457450    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.457482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.457476    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck299\" (UniqueName: \"kubernetes.io/projected/e9ec51bf-c382-404f-b380-3673166f0368-kube-api-access-ck299\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.457482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.457486    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9ec51bf-c382-404f-b380-3673166f0368-console-oauth-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.457683 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.457496    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:06.457683 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:06.457506    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec51bf-c382-404f-b380-3673166f0368-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:23:07.041250 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.041211    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9ec51bf-c382-404f-b380-3673166f0368" containerID="bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446" exitCode=0
Sep 08 13:23:07.041630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.041266    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffc67fd89-khspk"
Sep 08 13:23:07.041630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.041267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffc67fd89-khspk" event={"ID":"e9ec51bf-c382-404f-b380-3673166f0368","Type":"ContainerDied","Data":"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"}
Sep 08 13:23:07.041630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.041377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffc67fd89-khspk" event={"ID":"e9ec51bf-c382-404f-b380-3673166f0368","Type":"ContainerDied","Data":"3041c360ad9f30d9fea6955e7b241accb2ac30d3b8fa959abd3a7cbd11202bc4"}
Sep 08 13:23:07.041630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.041403    2580 scope.go:117] "RemoveContainer" containerID="bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"
Sep 08 13:23:07.049973 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.049949    2580 scope.go:117] "RemoveContainer" containerID="bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"
Sep 08 13:23:07.050265 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:23:07.050237    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446\": container with ID starting with bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446 not found: ID does not exist" containerID="bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"
Sep 08 13:23:07.050338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.050274    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446"} err="failed to get container status \"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446\": rpc error: code = NotFound desc = could not find container \"bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446\": container with ID starting with bd877c6e92afa5e0357022607ca4a9cb15c26301ac3294fb2d7a5528ae1b1446 not found: ID does not exist"
Sep 08 13:23:07.056968 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.056919    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:23:07.058812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:07.058786    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ffc67fd89-khspk"]
Sep 08 13:23:08.537013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:23:08.536978    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ec51bf-c382-404f-b380-3673166f0368" path="/var/lib/kubelet/pods/e9ec51bf-c382-404f-b380-3673166f0368/volumes"
Sep 08 13:24:27.391945 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.391864    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"]
Sep 08 13:24:27.392437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.392206    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ec51bf-c382-404f-b380-3673166f0368" containerName="console"
Sep 08 13:24:27.392437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.392219    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ec51bf-c382-404f-b380-3673166f0368" containerName="console"
Sep 08 13:24:27.392437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.392286    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ec51bf-c382-404f-b380-3673166f0368" containerName="console"
Sep 08 13:24:27.395234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.395217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.397930 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.397907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 13:24:27.399012 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.398991    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 13:24:27.399107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.399022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nbkgw\""
Sep 08 13:24:27.404949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.404922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"]
Sep 08 13:24:27.510438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.510402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.510606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.510461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.510606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.510498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvshv\" (UniqueName: \"kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.611495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.611462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.611669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.611510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvshv\" (UniqueName: \"kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.611669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.611580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.611868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.611848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.611932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.611885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.620445 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.620418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvshv\" (UniqueName: \"kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.704204 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.704180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:27.974983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.974909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"]
Sep 08 13:24:27.977850 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:24:27.977821    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac6ce62_fe01_4cb7_b3e9_0b2c6b854677.slice/crio-b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52 WatchSource:0}: Error finding container b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52: Status 404 returned error can't find the container with id b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52
Sep 08 13:24:27.979832 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:27.979813    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:24:28.358320 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:28.358236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx" event={"ID":"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677","Type":"ContainerStarted","Data":"b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52"}
Sep 08 13:24:34.384471 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:34.384430    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerID="431451a23a8566cc683c21932e94df431f2fd14be6c6377e8d4320f78bb4e1fc" exitCode=0
Sep 08 13:24:34.384844 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:34.384502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx" event={"ID":"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677","Type":"ContainerDied","Data":"431451a23a8566cc683c21932e94df431f2fd14be6c6377e8d4320f78bb4e1fc"}
Sep 08 13:24:37.398150 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:37.398112    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerID="290a1cc854f3ef6c69392741e8076b8f336b9544ec139b836d3d1d6e40f3450c" exitCode=0
Sep 08 13:24:37.398150 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:37.398151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx" event={"ID":"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677","Type":"ContainerDied","Data":"290a1cc854f3ef6c69392741e8076b8f336b9544ec139b836d3d1d6e40f3450c"}
Sep 08 13:24:45.435547 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:45.435507    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerID="58ba67f0a4f32983643e60b1bfb1d6a9251ec41da9c2418824b471fbb546dcb6" exitCode=0
Sep 08 13:24:45.435950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:45.435557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx" event={"ID":"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677","Type":"ContainerDied","Data":"58ba67f0a4f32983643e60b1bfb1d6a9251ec41da9c2418824b471fbb546dcb6"}
Sep 08 13:24:46.565734 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.565709    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:46.676114 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.676077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util\") pod \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") "
Sep 08 13:24:46.676336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.676128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle\") pod \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") "
Sep 08 13:24:46.676336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.676155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvshv\" (UniqueName: \"kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv\") pod \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\" (UID: \"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677\") "
Sep 08 13:24:46.680469 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.677266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle" (OuterVolumeSpecName: "bundle") pod "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" (UID: "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:46.680469 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.679227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv" (OuterVolumeSpecName: "kube-api-access-lvshv") pod "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" (UID: "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677"). InnerVolumeSpecName "kube-api-access-lvshv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:46.682773 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.682738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util" (OuterVolumeSpecName: "util") pod "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" (UID: "2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:46.777518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.777482    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-util\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:24:46.777518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.777515    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:24:46.777706 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:46.777547    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvshv\" (UniqueName: \"kubernetes.io/projected/2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677-kube-api-access-lvshv\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:24:47.445202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:47.445146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx" event={"ID":"2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677","Type":"ContainerDied","Data":"b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52"}
Sep 08 13:24:47.445202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:47.445188    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ap6jvx"
Sep 08 13:24:47.445404 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:47.445255    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4868cca7a78b0d6d40b4584ffb6e30106296a61989befb453265a2f1bd98c52"
Sep 08 13:24:54.192911 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.192873    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bdfcf65f8-th6b5"]
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193196    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="extract"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193208    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="extract"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193221    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="pull"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193227    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="pull"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="util"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="util"
Sep 08 13:24:54.193309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.193298    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac6ce62-fe01-4cb7-b3e9-0b2c6b854677" containerName="extract"
Sep 08 13:24:54.197707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.197688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.206959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.206936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdfcf65f8-th6b5"]
Sep 08 13:24:54.333562 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-oauth-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-service-ca\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-trusted-ca-bundle\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333849 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clp58\" (UniqueName: \"kubernetes.io/projected/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-kube-api-access-clp58\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.333849 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.333753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-oauth-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-service-ca\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-trusted-ca-bundle\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clp58\" (UniqueName: \"kubernetes.io/projected/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-kube-api-access-clp58\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-oauth-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-oauth-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.434588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.434545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.435076 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.435041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.435226 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.435150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-service-ca\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.435346 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.435325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-trusted-ca-bundle\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.435411 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.435388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-oauth-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.437058 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.437031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-serving-cert\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.437144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.437117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-console-oauth-config\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.445907 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.445855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clp58\" (UniqueName: \"kubernetes.io/projected/79e813c4-fc9c-4532-b9f8-0f9a7f8c4376-kube-api-access-clp58\") pod \"console-7bdfcf65f8-th6b5\" (UID: \"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376\") " pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.507680 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.507651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:24:54.783026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.782986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdfcf65f8-th6b5"]
Sep 08 13:24:54.785365 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:24:54.785333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e813c4_fc9c_4532_b9f8_0f9a7f8c4376.slice/crio-53853a26790567e401f680ea0a38c407c49893219d4c2223b4f80e0d045a27f6 WatchSource:0}: Error finding container 53853a26790567e401f680ea0a38c407c49893219d4c2223b4f80e0d045a27f6: Status 404 returned error can't find the container with id 53853a26790567e401f680ea0a38c407c49893219d4c2223b4f80e0d045a27f6
Sep 08 13:24:54.795890 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.795868    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-xn2x4"]
Sep 08 13:24:54.798485 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.798469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:54.801217 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 08 13:24:54.801320 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 08 13:24:54.801320 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801227    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 08 13:24:54.801544 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vk7qr\""
Sep 08 13:24:54.801649 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 08 13:24:54.801706 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.801663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 08 13:24:54.809196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.809151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-xn2x4"]
Sep 08 13:24:54.939047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.939007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzn4\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-kube-api-access-wzzn4\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:54.939264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.939066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:54.939264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:54.939104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/837f1a99-c4cf-4f76-9372-8cc748123563-cabundle0\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.039835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.039753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzn4\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-kube-api-access-wzzn4\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.039835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.039798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.039835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.039818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/837f1a99-c4cf-4f76-9372-8cc748123563-cabundle0\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.040186 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:24:55.039926    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 08 13:24:55.040186 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:24:55.039944    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 08 13:24:55.040186 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:24:55.039953    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-xn2x4: references non-existent secret key: ca.crt
Sep 08 13:24:55.040186 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:24:55.040002    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates podName:837f1a99-c4cf-4f76-9372-8cc748123563 nodeName:}" failed. No retries permitted until 2026-09-08 13:24:55.539987795 +0000 UTC m=+491.490641729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates") pod "keda-operator-579c8958f7-xn2x4" (UID: "837f1a99-c4cf-4f76-9372-8cc748123563") : references non-existent secret key: ca.crt
Sep 08 13:24:55.040501 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.040483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/837f1a99-c4cf-4f76-9372-8cc748123563-cabundle0\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.051292 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.051270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzn4\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-kube-api-access-wzzn4\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.478866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.478832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfcf65f8-th6b5" event={"ID":"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376","Type":"ContainerStarted","Data":"0df128019290c3c9b935960f048c694253f8ecb3f9e0e750e8b5352905f54eba"}
Sep 08 13:24:55.479249 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.478880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfcf65f8-th6b5" event={"ID":"79e813c4-fc9c-4532-b9f8-0f9a7f8c4376","Type":"ContainerStarted","Data":"53853a26790567e401f680ea0a38c407c49893219d4c2223b4f80e0d045a27f6"}
Sep 08 13:24:55.498881 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.498833    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bdfcf65f8-th6b5" podStartSLOduration=1.498813782 podStartE2EDuration="1.498813782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:55.498561523 +0000 UTC m=+491.449215539" watchObservedRunningTime="2026-09-08 13:24:55.498813782 +0000 UTC m=+491.449467735"
Sep 08 13:24:55.544482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.544446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.547252 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.547226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/837f1a99-c4cf-4f76-9372-8cc748123563-certificates\") pod \"keda-operator-579c8958f7-xn2x4\" (UID: \"837f1a99-c4cf-4f76-9372-8cc748123563\") " pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:55.765932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:55.765839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:56.041192 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:56.041153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-xn2x4"]
Sep 08 13:24:56.043670 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:24:56.043641    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837f1a99_c4cf_4f76_9372_8cc748123563.slice/crio-f5e3d358e8b764c29ec0ebf18a55cadea3aa49627f12c3b75a3e44a455e3daf1 WatchSource:0}: Error finding container f5e3d358e8b764c29ec0ebf18a55cadea3aa49627f12c3b75a3e44a455e3daf1: Status 404 returned error can't find the container with id f5e3d358e8b764c29ec0ebf18a55cadea3aa49627f12c3b75a3e44a455e3daf1
Sep 08 13:24:56.484696 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:56.484661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-xn2x4" event={"ID":"837f1a99-c4cf-4f76-9372-8cc748123563","Type":"ContainerStarted","Data":"f5e3d358e8b764c29ec0ebf18a55cadea3aa49627f12c3b75a3e44a455e3daf1"}
Sep 08 13:24:59.502509 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:59.502472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-xn2x4" event={"ID":"837f1a99-c4cf-4f76-9372-8cc748123563","Type":"ContainerStarted","Data":"d96671ad73dd8a73b3acdf60a41e617905a58bcb0a4a69d37c909a3f401d9d39"}
Sep 08 13:24:59.502937 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:59.502611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:24:59.521500 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:24:59.521455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-xn2x4" podStartSLOduration=2.615875134 podStartE2EDuration="5.521438209s" totalImagesPullingTime="2.905563075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:59.519746032 +0000 UTC m=+495.470399985" watchObservedRunningTime="2026-09-08 13:24:59.521438209 +0000 UTC m=+495.472092161"
Sep 08 13:25:04.507845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:04.507807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:25:04.508263 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:04.507917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:25:04.512613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:04.512589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:25:04.529530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:04.529508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bdfcf65f8-th6b5"
Sep 08 13:25:04.586988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:04.586961    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:25:20.510654 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:20.510621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-xn2x4"
Sep 08 13:25:29.606342 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.606281    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68d5d99b7b-8w2qk" podUID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" containerName="console" containerID="cri-o://905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0" gracePeriod=15
Sep 08 13:25:29.839234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.839209    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:25:29.938066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.937979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl99d\" (UniqueName: \"kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938366 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938366 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938366 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938366 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle\") pod \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\" (UID: \"19a571f7-e83a-4cd8-91f3-bf8576d85a02\") "
Sep 08 13:25:29.938585 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config" (OuterVolumeSpecName: "console-config") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:25:29.938585 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca" (OuterVolumeSpecName: "service-ca") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:25:29.938585 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:25:29.938713 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938639    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-oauth-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:29.938713 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938659    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:29.938713 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938676    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:29.938944 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.938922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:25:29.940290 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.940258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:25:29.940386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.940287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d" (OuterVolumeSpecName: "kube-api-access-bl99d") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "kube-api-access-bl99d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:25:29.940386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:29.940342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19a571f7-e83a-4cd8-91f3-bf8576d85a02" (UID: "19a571f7-e83a-4cd8-91f3-bf8576d85a02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:25:30.039414 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.039386    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:30.039414 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.039409    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl99d\" (UniqueName: \"kubernetes.io/projected/19a571f7-e83a-4cd8-91f3-bf8576d85a02-kube-api-access-bl99d\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:30.039414 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.039419    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a571f7-e83a-4cd8-91f3-bf8576d85a02-console-oauth-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:30.039586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.039427    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19a571f7-e83a-4cd8-91f3-bf8576d85a02-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:25:30.639228 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.639193    2580 generic.go:356] "Generic (PLEG): container finished" podID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" containerID="905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0" exitCode=0
Sep 08 13:25:30.639616 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.639266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d5d99b7b-8w2qk" event={"ID":"19a571f7-e83a-4cd8-91f3-bf8576d85a02","Type":"ContainerDied","Data":"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"}
Sep 08 13:25:30.639616 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.639278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d5d99b7b-8w2qk"
Sep 08 13:25:30.639616 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.639301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d5d99b7b-8w2qk" event={"ID":"19a571f7-e83a-4cd8-91f3-bf8576d85a02","Type":"ContainerDied","Data":"dc52cbe95be661e8585fc8f7a2a69675842f0bfd7f6cbc670889ee266f23da48"}
Sep 08 13:25:30.639616 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.639320    2580 scope.go:117] "RemoveContainer" containerID="905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"
Sep 08 13:25:30.647702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.647683    2580 scope.go:117] "RemoveContainer" containerID="905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"
Sep 08 13:25:30.647991 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:25:30.647970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0\": container with ID starting with 905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0 not found: ID does not exist" containerID="905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"
Sep 08 13:25:30.648037 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.647999    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0"} err="failed to get container status \"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0\": rpc error: code = NotFound desc = could not find container \"905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0\": container with ID starting with 905e8fc21f4e4270a8a3a4e2234dce803deb8de4fe858bae8379ad5dc86702c0 not found: ID does not exist"
Sep 08 13:25:30.655868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.655847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:25:30.660100 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:30.660078    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68d5d99b7b-8w2qk"]
Sep 08 13:25:32.534833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:25:32.534792    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" path="/var/lib/kubelet/pods/19a571f7-e83a-4cd8-91f3-bf8576d85a02/volumes"
Sep 08 13:26:13.193019 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.192984    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"]
Sep 08 13:26:13.193505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.193324    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" containerName="console"
Sep 08 13:26:13.193505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.193336    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" containerName="console"
Sep 08 13:26:13.193505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.193396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a571f7-e83a-4cd8-91f3-bf8576d85a02" containerName="console"
Sep 08 13:26:13.196021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.196003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.198569 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.198547    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-bqfvh\""
Sep 08 13:26:13.198687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.198610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 08 13:26:13.198687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.198547    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:26:13.199463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.199446    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:26:13.207265 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.207240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"]
Sep 08 13:26:13.240363 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.240330    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-hqctt"]
Sep 08 13:26:13.243507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.243490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.246224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.246206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 08 13:26:13.246306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.246207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4vj2j\""
Sep 08 13:26:13.252289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.252262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-hqctt"]
Sep 08 13:26:13.298846 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.298820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afae64f-7fc3-409a-9408-80bf2cd99ada-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.298986 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.298852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnms\" (UniqueName: \"kubernetes.io/projected/1afae64f-7fc3-409a-9408-80bf2cd99ada-kube-api-access-njnms\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.399779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.399742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afae64f-7fc3-409a-9408-80bf2cd99ada-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.399779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.399782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5t5\" (UniqueName: \"kubernetes.io/projected/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-kube-api-access-qn5t5\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.399985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.399811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnms\" (UniqueName: \"kubernetes.io/projected/1afae64f-7fc3-409a-9408-80bf2cd99ada-kube-api-access-njnms\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.399985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.399833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-data\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.402221 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.402198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1afae64f-7fc3-409a-9408-80bf2cd99ada-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.407833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.407813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnms\" (UniqueName: \"kubernetes.io/projected/1afae64f-7fc3-409a-9408-80bf2cd99ada-kube-api-access-njnms\") pod \"llmisvc-controller-manager-6f9bbc7b7c-wd56h\" (UID: \"1afae64f-7fc3-409a-9408-80bf2cd99ada\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.501159 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.501123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5t5\" (UniqueName: \"kubernetes.io/projected/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-kube-api-access-qn5t5\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.501368 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.501190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-data\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.501584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.501563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-data\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.507063 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.507045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:13.510309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.510232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5t5\" (UniqueName: \"kubernetes.io/projected/e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec-kube-api-access-qn5t5\") pod \"seaweedfs-64568bcd49-hqctt\" (UID: \"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec\") " pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.553444 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.553406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:13.777489 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.777408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"]
Sep 08 13:26:13.780152 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:26:13.780125    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1afae64f_7fc3_409a_9408_80bf2cd99ada.slice/crio-5c7f53db418ca9934584b8576a4982597d7eb3e69f244fd001ae0b988bc5f68b WatchSource:0}: Error finding container 5c7f53db418ca9934584b8576a4982597d7eb3e69f244fd001ae0b988bc5f68b: Status 404 returned error can't find the container with id 5c7f53db418ca9934584b8576a4982597d7eb3e69f244fd001ae0b988bc5f68b
Sep 08 13:26:13.818564 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.818538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h" event={"ID":"1afae64f-7fc3-409a-9408-80bf2cd99ada","Type":"ContainerStarted","Data":"5c7f53db418ca9934584b8576a4982597d7eb3e69f244fd001ae0b988bc5f68b"}
Sep 08 13:26:13.829034 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:13.829012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-hqctt"]
Sep 08 13:26:13.831305 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:26:13.831283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05c3d3d_5b29_43a7_9bcc_86bf2e3db6ec.slice/crio-d8525987fc5d600561913d1d77614f4f6eb6f922e02aa976d82a0173f55726d3 WatchSource:0}: Error finding container d8525987fc5d600561913d1d77614f4f6eb6f922e02aa976d82a0173f55726d3: Status 404 returned error can't find the container with id d8525987fc5d600561913d1d77614f4f6eb6f922e02aa976d82a0173f55726d3
Sep 08 13:26:14.824911 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:14.824880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-hqctt" event={"ID":"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec","Type":"ContainerStarted","Data":"d8525987fc5d600561913d1d77614f4f6eb6f922e02aa976d82a0173f55726d3"}
Sep 08 13:26:17.846676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:17.846640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-hqctt" event={"ID":"e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec","Type":"ContainerStarted","Data":"e901ee9f298e83b90fa8c6ad709e740809b09fee32717a1be966069c57484ddc"}
Sep 08 13:26:17.847135 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:17.846772    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:17.867482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:17.867425    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-hqctt" podStartSLOduration=1.554216179 podStartE2EDuration="4.86739904s" totalImagesPullingTime="3.313182861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:17.865398786 +0000 UTC m=+573.816052740" watchObservedRunningTime="2026-09-08 13:26:17.86739904 +0000 UTC m=+573.818052992"
Sep 08 13:26:19.856545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:19.856510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h" event={"ID":"1afae64f-7fc3-409a-9408-80bf2cd99ada","Type":"ContainerStarted","Data":"6cd75725072b241537bcb50094078c74c9602d7913155ebc1337a62c8dc589d3"}
Sep 08 13:26:19.856951 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:19.856569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:26:19.875371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:19.875319    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h" podStartSLOduration=1.8695142169999999 podStartE2EDuration="6.875301651s" totalImagesPullingTime="5.005787434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:19.874452966 +0000 UTC m=+575.825106919" watchObservedRunningTime="2026-09-08 13:26:19.875301651 +0000 UTC m=+575.825955602"
Sep 08 13:26:23.853187 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:23.853129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-hqctt"
Sep 08 13:26:44.456899 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:44.456871    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:26:44.457419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:44.456871    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:26:50.863160 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:26:50.863129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-wd56h"
Sep 08 13:27:32.527924 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.527889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-x8pjd"]
Sep 08 13:27:32.531302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.531279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.534340 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.534311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 08 13:27:32.534470 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.534407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4p8gh\""
Sep 08 13:27:32.537513 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.537493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ad32-b8cd-45d2-8875-5d329a73ec07-tls-certs\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.537613 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.537586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xktj\" (UniqueName: \"kubernetes.io/projected/3d37ad32-b8cd-45d2-8875-5d329a73ec07-kube-api-access-8xktj\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.546902 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.546876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x8pjd"]
Sep 08 13:27:32.638780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.638745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xktj\" (UniqueName: \"kubernetes.io/projected/3d37ad32-b8cd-45d2-8875-5d329a73ec07-kube-api-access-8xktj\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.638996 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.638962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ad32-b8cd-45d2-8875-5d329a73ec07-tls-certs\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.641448 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.641421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ad32-b8cd-45d2-8875-5d329a73ec07-tls-certs\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.651023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.650992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xktj\" (UniqueName: \"kubernetes.io/projected/3d37ad32-b8cd-45d2-8875-5d329a73ec07-kube-api-access-8xktj\") pod \"model-serving-api-7b954ffc95-x8pjd\" (UID: \"3d37ad32-b8cd-45d2-8875-5d329a73ec07\") " pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:32.842278 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:32.842194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:33.115784 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:33.115755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x8pjd"]
Sep 08 13:27:33.118295 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:27:33.118266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d37ad32_b8cd_45d2_8875_5d329a73ec07.slice/crio-50f27e077d5eaeb177b9a77afee59793ede5675bc8e1781937b91ded969ebb44 WatchSource:0}: Error finding container 50f27e077d5eaeb177b9a77afee59793ede5675bc8e1781937b91ded969ebb44: Status 404 returned error can't find the container with id 50f27e077d5eaeb177b9a77afee59793ede5675bc8e1781937b91ded969ebb44
Sep 08 13:27:33.169599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:33.169567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x8pjd" event={"ID":"3d37ad32-b8cd-45d2-8875-5d329a73ec07","Type":"ContainerStarted","Data":"50f27e077d5eaeb177b9a77afee59793ede5675bc8e1781937b91ded969ebb44"}
Sep 08 13:27:35.182071 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:35.182035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x8pjd" event={"ID":"3d37ad32-b8cd-45d2-8875-5d329a73ec07","Type":"ContainerStarted","Data":"ce6f06d1adc5feb8669d585d25e9d7eeb8ea5cd21b88b009bf3b9d055dd52dec"}
Sep 08 13:27:35.182462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:35.182105    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:35.201032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:35.200982    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-x8pjd" podStartSLOduration=1.8768381330000001 podStartE2EDuration="3.200967966s" totalImagesPullingTime="1.324129833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:35.200843208 +0000 UTC m=+651.151497171" watchObservedRunningTime="2026-09-08 13:27:35.200967966 +0000 UTC m=+651.151621918"
Sep 08 13:27:46.191323 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:46.191290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-x8pjd"
Sep 08 13:27:49.126465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.126430    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-j5p7m"]
Sep 08 13:27:49.129768 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.129752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.133861 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.133842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:27:49.155882 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.155860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j5p7m"]
Sep 08 13:27:49.179514 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.179485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5mm\" (UniqueName: \"kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.179614 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.179596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.280007 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.279977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.280152 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.280028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5mm\" (UniqueName: \"kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.280356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.280336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.289986 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.289964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5mm\" (UniqueName: \"kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm\") pod \"s3-init-j5p7m\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") " pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.453557 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.453531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j5p7m"
Sep 08 13:27:49.724429 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:49.724360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j5p7m"]
Sep 08 13:27:50.251871 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:50.251837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerStarted","Data":"8e32504fd647e6605710f8e39ed8915a79b6ceb72396446a5a91647367abf9c3"}
Sep 08 13:27:56.285977 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:56.285932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerStarted","Data":"cdf75102879dc1eaaec5cb49bf1e95dcc44918f57c7d68276b6181ec60e07c05"}
Sep 08 13:27:58.297251 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:58.297216    2580 generic.go:356] "Generic (PLEG): container finished" podID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerID="cdf75102879dc1eaaec5cb49bf1e95dcc44918f57c7d68276b6181ec60e07c05" exitCode=0
Sep 08 13:27:58.297640 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:27:58.297286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerDied","Data":"cdf75102879dc1eaaec5cb49bf1e95dcc44918f57c7d68276b6181ec60e07c05"}
Sep 08 13:28:03.323917 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:28:03.323878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerStarted","Data":"e6078fdd7824f715d8320cbcb9368232ee3caf0604d104fe894121c1908ba901"}
Sep 08 13:28:03.344994 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:28:03.344934    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-j5p7m" podStartSLOduration=2.060498277 podStartE2EDuration="14.344913754s" totalImagesPullingTime="10.753237611s" totalInitContainerRuntime="1.531177866s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:03.34284594 +0000 UTC m=+679.293499894" watchObservedRunningTime="2026-09-08 13:28:03.344913754 +0000 UTC m=+679.295567708"
Sep 08 13:29:08.605199 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:08.605141    2580 generic.go:356] "Generic (PLEG): container finished" podID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerID="e6078fdd7824f715d8320cbcb9368232ee3caf0604d104fe894121c1908ba901" exitCode=0
Sep 08 13:29:08.605634 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:08.605203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerDied","Data":"e6078fdd7824f715d8320cbcb9368232ee3caf0604d104fe894121c1908ba901"}
Sep 08 13:29:09.736333 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.736305    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j5p7m"
Sep 08 13:29:09.767055 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.767026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache\") pod \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") "
Sep 08 13:29:09.767186 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.767120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd5mm\" (UniqueName: \"kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm\") pod \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\" (UID: \"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9\") "
Sep 08 13:29:09.769158 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.769133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm" (OuterVolumeSpecName: "kube-api-access-rd5mm") pod "c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" (UID: "c6b91ae4-b98b-422b-bfdd-1d0b020e01b9"). InnerVolumeSpecName "kube-api-access-rd5mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:29:09.789020 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.788996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" (UID: "c6b91ae4-b98b-422b-bfdd-1d0b020e01b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:29:09.868406 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.868326    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd5mm\" (UniqueName: \"kubernetes.io/projected/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-kube-api-access-rd5mm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:09.868406 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:09.868354    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6b91ae4-b98b-422b-bfdd-1d0b020e01b9-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:10.614849 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:10.614815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j5p7m" event={"ID":"c6b91ae4-b98b-422b-bfdd-1d0b020e01b9","Type":"ContainerDied","Data":"8e32504fd647e6605710f8e39ed8915a79b6ceb72396446a5a91647367abf9c3"}
Sep 08 13:29:10.614983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:10.614863    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j5p7m"
Sep 08 13:29:10.614983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:10.614860    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e32504fd647e6605710f8e39ed8915a79b6ceb72396446a5a91647367abf9c3"
Sep 08 13:29:11.362956 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.362925    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:11.363337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.363302    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerName="download-hf-model"
Sep 08 13:29:11.363337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.363314    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerName="download-hf-model"
Sep 08 13:29:11.363337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.363322    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerName="s3-init"
Sep 08 13:29:11.363337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.363327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerName="s3-init"
Sep 08 13:29:11.363461 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.363394    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b91ae4-b98b-422b-bfdd-1d0b020e01b9" containerName="s3-init"
Sep 08 13:29:11.366143 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.366127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.368510 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.368486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 08 13:29:11.376156 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.376128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:11.379077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.379050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjx2v\" (UniqueName: \"kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.379185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.379108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.479768 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.479737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjx2v\" (UniqueName: \"kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.479916 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.479787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.480185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.480146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.489439 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.489420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjx2v\" (UniqueName: \"kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v\") pod \"seaweedfs-tls-custom-6644bf5d6c-9npfb\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.675732 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.675654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:11.945190 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:11.945057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:11.947944 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:11.947913    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4155b6_71e4_4707_87ef_1d0eb5bc2e5b.slice/crio-b5e1ad1b48156fb5b7acb289c1ef035f87eccfa6fc2076dd93952e55dd14929e WatchSource:0}: Error finding container b5e1ad1b48156fb5b7acb289c1ef035f87eccfa6fc2076dd93952e55dd14929e: Status 404 returned error can't find the container with id b5e1ad1b48156fb5b7acb289c1ef035f87eccfa6fc2076dd93952e55dd14929e
Sep 08 13:29:12.625381 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:12.625301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" event={"ID":"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b","Type":"ContainerStarted","Data":"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"}
Sep 08 13:29:12.625381 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:12.625348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" event={"ID":"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b","Type":"ContainerStarted","Data":"b5e1ad1b48156fb5b7acb289c1ef035f87eccfa6fc2076dd93952e55dd14929e"}
Sep 08 13:29:12.643991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:12.643941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" podStartSLOduration=1.339366472 podStartE2EDuration="1.643925414s" totalImagesPullingTime="304.558942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:12.642358343 +0000 UTC m=+748.593012296" watchObservedRunningTime="2026-09-08 13:29:12.643925414 +0000 UTC m=+748.594579366"
Sep 08 13:29:13.974525 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:13.974486    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:14.634252 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:14.634210    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" podUID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" containerName="seaweedfs-tls-custom" containerID="cri-o://6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f" gracePeriod=30
Sep 08 13:29:15.873438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:15.873412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:15.914072 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:15.913986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjx2v\" (UniqueName: \"kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v\") pod \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") "
Sep 08 13:29:15.914072 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:15.914022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data\") pod \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\" (UID: \"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b\") "
Sep 08 13:29:15.915897 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:15.915865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data" (OuterVolumeSpecName: "data") pod "5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" (UID: "5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:29:15.916097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:15.916070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v" (OuterVolumeSpecName: "kube-api-access-qjx2v") pod "5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" (UID: "5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b"). InnerVolumeSpecName "kube-api-access-qjx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:29:16.014956 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.014906    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjx2v\" (UniqueName: \"kubernetes.io/projected/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-kube-api-access-qjx2v\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:16.014956 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.014948    2580 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b-data\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:16.645535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.645503    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" containerID="6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f" exitCode=0
Sep 08 13:29:16.645698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.645543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"
Sep 08 13:29:16.645698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.645555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" event={"ID":"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b","Type":"ContainerDied","Data":"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"}
Sep 08 13:29:16.645698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.645593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb" event={"ID":"5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b","Type":"ContainerDied","Data":"b5e1ad1b48156fb5b7acb289c1ef035f87eccfa6fc2076dd93952e55dd14929e"}
Sep 08 13:29:16.645698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.645615    2580 scope.go:117] "RemoveContainer" containerID="6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"
Sep 08 13:29:16.655266 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.655247    2580 scope.go:117] "RemoveContainer" containerID="6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"
Sep 08 13:29:16.655547 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:29:16.655528    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f\": container with ID starting with 6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f not found: ID does not exist" containerID="6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"
Sep 08 13:29:16.655602 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.655558    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f"} err="failed to get container status \"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f\": rpc error: code = NotFound desc = could not find container \"6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f\": container with ID starting with 6682cf64ca1ebf66a47a5cf1b4de3f9352274b817635f1a1e449fb1f8739402f not found: ID does not exist"
Sep 08 13:29:16.661791 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.661770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:16.667761 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:16.667742    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9npfb"]
Sep 08 13:29:18.535330 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:18.535294    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" path="/var/lib/kubelet/pods/5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b/volumes"
Sep 08 13:29:21.052584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.052546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6ldf9"]
Sep 08 13:29:21.052959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.052880    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" containerName="seaweedfs-tls-custom"
Sep 08 13:29:21.052959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.052892    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" containerName="seaweedfs-tls-custom"
Sep 08 13:29:21.052959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.052954    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f4155b6-71e4-4707-87ef-1d0eb5bc2e5b" containerName="seaweedfs-tls-custom"
Sep 08 13:29:21.057488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.057468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.060449 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.060433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:29:21.060759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.060745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 08 13:29:21.066445 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.066425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6ldf9"]
Sep 08 13:29:21.148371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.148337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjtx\" (UniqueName: \"kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.148551 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.148384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.248919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.248882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjtx\" (UniqueName: \"kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.249088 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.248930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.249399 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.249375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.259112 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.259089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjtx\" (UniqueName: \"kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx\") pod \"s3-tls-init-custom-6ldf9\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") " pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.381717 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.381637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:21.651025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.650998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6ldf9"]
Sep 08 13:29:21.653313 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:21.653283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ea7b35_3f02_4031_8956_b2461782a408.slice/crio-3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445 WatchSource:0}: Error finding container 3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445: Status 404 returned error can't find the container with id 3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445
Sep 08 13:29:21.675897 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:21.675871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerStarted","Data":"3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445"}
Sep 08 13:29:22.683911 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:22.683869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerStarted","Data":"b3969a3c54158652dd43cb80c915c1a2cbc205fc5e9ba7ca8b3f1e890f0c5135"}
Sep 08 13:29:23.689714 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:23.689622    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6ea7b35-3f02-4031-8956-b2461782a408" containerID="b3969a3c54158652dd43cb80c915c1a2cbc205fc5e9ba7ca8b3f1e890f0c5135" exitCode=0
Sep 08 13:29:23.690073 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:23.689714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerDied","Data":"b3969a3c54158652dd43cb80c915c1a2cbc205fc5e9ba7ca8b3f1e890f0c5135"}
Sep 08 13:29:24.695282 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:24.695247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerStarted","Data":"bf5feb71b31d6c2a42e6600b8c8bfd4cf3380168fbcaa66b9472c76d4d905a06"}
Sep 08 13:29:24.715316 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:24.715265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6ldf9" podStartSLOduration=2.049257643 podStartE2EDuration="3.71524663s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.665988987s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:24.713904491 +0000 UTC m=+760.664558443" watchObservedRunningTime="2026-09-08 13:29:24.71524663 +0000 UTC m=+760.665900582"
Sep 08 13:29:30.729179 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:30.729129    2580 generic.go:356] "Generic (PLEG): container finished" podID="f6ea7b35-3f02-4031-8956-b2461782a408" containerID="bf5feb71b31d6c2a42e6600b8c8bfd4cf3380168fbcaa66b9472c76d4d905a06" exitCode=0
Sep 08 13:29:30.729546 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:30.729205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerDied","Data":"bf5feb71b31d6c2a42e6600b8c8bfd4cf3380168fbcaa66b9472c76d4d905a06"}
Sep 08 13:29:31.871579 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:31.871557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:31.929107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:31.929078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache\") pod \"f6ea7b35-3f02-4031-8956-b2461782a408\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") "
Sep 08 13:29:31.929281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:31.929144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdjtx\" (UniqueName: \"kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx\") pod \"f6ea7b35-3f02-4031-8956-b2461782a408\" (UID: \"f6ea7b35-3f02-4031-8956-b2461782a408\") "
Sep 08 13:29:31.931125 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:31.931093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx" (OuterVolumeSpecName: "kube-api-access-pdjtx") pod "f6ea7b35-3f02-4031-8956-b2461782a408" (UID: "f6ea7b35-3f02-4031-8956-b2461782a408"). InnerVolumeSpecName "kube-api-access-pdjtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:29:31.951801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:31.951776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6ea7b35-3f02-4031-8956-b2461782a408" (UID: "f6ea7b35-3f02-4031-8956-b2461782a408"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:29:32.030665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:32.030595    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea7b35-3f02-4031-8956-b2461782a408-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:32.030665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:32.030623    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjtx\" (UniqueName: \"kubernetes.io/projected/f6ea7b35-3f02-4031-8956-b2461782a408-kube-api-access-pdjtx\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:32.739184 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:32.739145    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6ldf9"
Sep 08 13:29:32.739329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:32.739186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6ldf9" event={"ID":"f6ea7b35-3f02-4031-8956-b2461782a408","Type":"ContainerDied","Data":"3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445"}
Sep 08 13:29:32.739329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:32.739224    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dea2e3226297e2e499feeebc52e7d40b575c3591f29986f88af2eab69814445"
Sep 08 13:29:33.355292 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"]
Sep 08 13:29:33.355651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355583    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ea7b35-3f02-4031-8956-b2461782a408" containerName="s3-tls-init-custom"
Sep 08 13:29:33.355651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355602    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ea7b35-3f02-4031-8956-b2461782a408" containerName="s3-tls-init-custom"
Sep 08 13:29:33.355651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ea7b35-3f02-4031-8956-b2461782a408" containerName="download-hf-model"
Sep 08 13:29:33.355651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ea7b35-3f02-4031-8956-b2461782a408" containerName="download-hf-model"
Sep 08 13:29:33.355780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.355675    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ea7b35-3f02-4031-8956-b2461782a408" containerName="s3-tls-init-custom"
Sep 08 13:29:33.359625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.359609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.362334 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.362313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 08 13:29:33.362443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.362359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 08 13:29:33.367582 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.367559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"]
Sep 08 13:29:33.441800 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.441767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.441939 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.441830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5k2\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-kube-api-access-kk5k2\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.441939 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.441896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.542415 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.542382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5k2\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-kube-api-access-kk5k2\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.542582 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.542427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.542582 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.542479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.542829 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.542808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.544871 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.544841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.551700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.551674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5k2\" (UniqueName: \"kubernetes.io/projected/5f0c840e-b02a-45d8-b7dc-d9d99183a43c-kube-api-access-kk5k2\") pod \"seaweedfs-tls-serving-67b7477c6d-sth9g\" (UID: \"5f0c840e-b02a-45d8-b7dc-d9d99183a43c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.668732 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.668634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"
Sep 08 13:29:33.940119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.940093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-sth9g"]
Sep 08 13:29:33.942186 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:33.942147    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0c840e_b02a_45d8_b7dc_d9d99183a43c.slice/crio-17cfd698b22cf8b7d4c371061c9d3c676b3139c432bf956e6e1a24c6e3879902 WatchSource:0}: Error finding container 17cfd698b22cf8b7d4c371061c9d3c676b3139c432bf956e6e1a24c6e3879902: Status 404 returned error can't find the container with id 17cfd698b22cf8b7d4c371061c9d3c676b3139c432bf956e6e1a24c6e3879902
Sep 08 13:29:33.943521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:33.943503    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:29:34.751356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:34.751311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g" event={"ID":"5f0c840e-b02a-45d8-b7dc-d9d99183a43c","Type":"ContainerStarted","Data":"51d0862afad74522d981f16e92bca8024bae9541ed24f477fbaabf808a4bd447"}
Sep 08 13:29:34.751782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:34.751402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g" event={"ID":"5f0c840e-b02a-45d8-b7dc-d9d99183a43c","Type":"ContainerStarted","Data":"17cfd698b22cf8b7d4c371061c9d3c676b3139c432bf956e6e1a24c6e3879902"}
Sep 08 13:29:34.771194 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:34.771132    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-sth9g" podStartSLOduration=1.5199299370000001 podStartE2EDuration="1.77111708s" totalImagesPullingTime="251.187143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:34.769115068 +0000 UTC m=+770.719769020" watchObservedRunningTime="2026-09-08 13:29:34.77111708 +0000 UTC m=+770.721771029"
Sep 08 13:29:35.318012 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.317974    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-tqs7n"]
Sep 08 13:29:35.322635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.322615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.325105 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.325083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:29:35.332180 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.332141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-tqs7n"]
Sep 08 13:29:35.358066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.358040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.358223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.358106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmcmj\" (UniqueName: \"kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.458602 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.458566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmcmj\" (UniqueName: \"kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.458733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.458623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.458992 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.458972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.468122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.468099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmcmj\" (UniqueName: \"kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj\") pod \"s3-tls-init-serving-tqs7n\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") " pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.641050 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.640965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:35.911684 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:35.911573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-tqs7n"]
Sep 08 13:29:35.914109 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:35.914083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod392ae552_d454_4599_8d6a_b88a306b77c2.slice/crio-59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f WatchSource:0}: Error finding container 59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f: Status 404 returned error can't find the container with id 59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f
Sep 08 13:29:36.763323 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:36.763133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerStarted","Data":"53cbb24cf3920351a84a48f3f7a4357a06e7febf44d5eb3ae1110f39951fe9ec"}
Sep 08 13:29:36.763323 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:36.763211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerStarted","Data":"59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f"}
Sep 08 13:29:37.769490 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:37.769457    2580 generic.go:356] "Generic (PLEG): container finished" podID="392ae552-d454-4599-8d6a-b88a306b77c2" containerID="53cbb24cf3920351a84a48f3f7a4357a06e7febf44d5eb3ae1110f39951fe9ec" exitCode=0
Sep 08 13:29:37.769862 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:37.769507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerDied","Data":"53cbb24cf3920351a84a48f3f7a4357a06e7febf44d5eb3ae1110f39951fe9ec"}
Sep 08 13:29:38.775725 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:38.775682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerStarted","Data":"b7c3f32256e1528887216373d9a9e9a89712d3d753699f02f5d51738bd9ac0d7"}
Sep 08 13:29:38.794901 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:38.794846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-tqs7n" podStartSLOduration=2.1666506229999998 podStartE2EDuration="3.794830682s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.628180059s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:38.793472978 +0000 UTC m=+774.744126931" watchObservedRunningTime="2026-09-08 13:29:38.794830682 +0000 UTC m=+774.745484634"
Sep 08 13:29:43.801130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:43.801097    2580 generic.go:356] "Generic (PLEG): container finished" podID="392ae552-d454-4599-8d6a-b88a306b77c2" containerID="b7c3f32256e1528887216373d9a9e9a89712d3d753699f02f5d51738bd9ac0d7" exitCode=0
Sep 08 13:29:43.801493 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:43.801158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerDied","Data":"b7c3f32256e1528887216373d9a9e9a89712d3d753699f02f5d51738bd9ac0d7"}
Sep 08 13:29:44.935781 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:44.935759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:45.039531 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.039497    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache\") pod \"392ae552-d454-4599-8d6a-b88a306b77c2\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") "
Sep 08 13:29:45.039531 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.039533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmcmj\" (UniqueName: \"kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj\") pod \"392ae552-d454-4599-8d6a-b88a306b77c2\" (UID: \"392ae552-d454-4599-8d6a-b88a306b77c2\") "
Sep 08 13:29:45.041674 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.041638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj" (OuterVolumeSpecName: "kube-api-access-kmcmj") pod "392ae552-d454-4599-8d6a-b88a306b77c2" (UID: "392ae552-d454-4599-8d6a-b88a306b77c2"). InnerVolumeSpecName "kube-api-access-kmcmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:29:45.061076 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.061014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "392ae552-d454-4599-8d6a-b88a306b77c2" (UID: "392ae552-d454-4599-8d6a-b88a306b77c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:29:45.141203 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.141152    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/392ae552-d454-4599-8d6a-b88a306b77c2-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:45.141203 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.141197    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmcmj\" (UniqueName: \"kubernetes.io/projected/392ae552-d454-4599-8d6a-b88a306b77c2-kube-api-access-kmcmj\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:29:45.812060 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.812031    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-tqs7n"
Sep 08 13:29:45.812257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.812034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-tqs7n" event={"ID":"392ae552-d454-4599-8d6a-b88a306b77c2","Type":"ContainerDied","Data":"59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f"}
Sep 08 13:29:45.812257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:45.812143    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59bbaf65d9f64e682273ef1cbef0e5768159f9809f50f3ccb64d7c804558368f"
Sep 08 13:29:58.052064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052023    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:29:58.052543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052529    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="392ae552-d454-4599-8d6a-b88a306b77c2" containerName="download-hf-model"
Sep 08 13:29:58.052615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052551    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="392ae552-d454-4599-8d6a-b88a306b77c2" containerName="download-hf-model"
Sep 08 13:29:58.052615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="392ae552-d454-4599-8d6a-b88a306b77c2" containerName="s3-tls-init-serving"
Sep 08 13:29:58.052615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052586    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="392ae552-d454-4599-8d6a-b88a306b77c2" containerName="s3-tls-init-serving"
Sep 08 13:29:58.052761 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.052686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="392ae552-d454-4599-8d6a-b88a306b77c2" containerName="s3-tls-init-serving"
Sep 08 13:29:58.055026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.055003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.057881 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.057854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:29:58.058021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.057854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 08 13:29:58.058021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.057857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:29:58.058021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.057965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:29:58.058209 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.058124    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:29:58.058458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.058444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 08 13:29:58.058641 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.058628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:29:58.067313 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.067292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:29:58.146673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.146634    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:29:58.148965 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.148939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.149088 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.149012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.149154 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.149101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.149154 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.149130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlzm\" (UniqueName: \"kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.149261 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.149200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.149310 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.149268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.151853 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.151826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f54a2b\""
Sep 08 13:29:58.151853 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.151833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f54a2b-kube-rbac-proxy-sar-config\""
Sep 08 13:29:58.152045 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.151862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f54a2b-dockercfg-6kqb7\""
Sep 08 13:29:58.152045 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.151850    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f54a2b-predictor-serving-cert\""
Sep 08 13:29:58.165528 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.165505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:29:58.250507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlzm\" (UniqueName: \"kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.250698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd859\" (UniqueName: \"kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.251036 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.251036 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.250796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.251036 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:29:58.250886    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 08 13:29:58.251036 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:29:58.250967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls podName:dc3b8fad-2b03-457f-85bf-564719775fe5 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:58.750944935 +0000 UTC m=+794.701598879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls") pod "isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 08 13:29:58.251269 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.251071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.251269 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.251262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.251338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.251264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.258762 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.258741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlzm\" (UniqueName: \"kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.352013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.351945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.351981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd859\" (UniqueName: \"kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352284 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:29:58.352110    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:29:58.352284 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:29:58.352227    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:58.852206657 +0000 UTC m=+794.802860602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:29:58.352540 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352679 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.352779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.352760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.362298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.362274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd859\" (UniqueName: \"kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.756547 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.756507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.759059 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.759038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:58.857902 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.857871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.860223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.860194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") pod \"isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:58.965395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:58.965355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:29:59.059593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.059552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:29:59.241314 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.241289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:29:59.243112 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:59.243084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3b8fad_2b03_457f_85bf_564719775fe5.slice/crio-7dc92dc8db426279b66022c18f5bd7c489ca7302f4f0de10e6d9780912f93e26 WatchSource:0}: Error finding container 7dc92dc8db426279b66022c18f5bd7c489ca7302f4f0de10e6d9780912f93e26: Status 404 returned error can't find the container with id 7dc92dc8db426279b66022c18f5bd7c489ca7302f4f0de10e6d9780912f93e26
Sep 08 13:29:59.331401 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.331377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:29:59.333331 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:29:59.333304    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9bad24_c0d9_4e1d_bda5_32c56a79b9e8.slice/crio-1c21dcc9c87f3b638e5dfdd131b271a4ee3df6f835af86f780202985cb67540d WatchSource:0}: Error finding container 1c21dcc9c87f3b638e5dfdd131b271a4ee3df6f835af86f780202985cb67540d: Status 404 returned error can't find the container with id 1c21dcc9c87f3b638e5dfdd131b271a4ee3df6f835af86f780202985cb67540d
Sep 08 13:29:59.881432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.881391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerStarted","Data":"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"}
Sep 08 13:29:59.881632 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.881474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerStarted","Data":"7dc92dc8db426279b66022c18f5bd7c489ca7302f4f0de10e6d9780912f93e26"}
Sep 08 13:29:59.883465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.883436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerStarted","Data":"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"}
Sep 08 13:29:59.883605 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:29:59.883500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerStarted","Data":"1c21dcc9c87f3b638e5dfdd131b271a4ee3df6f835af86f780202985cb67540d"}
Sep 08 13:30:01.895099 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:01.895073    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/0.log"
Sep 08 13:30:01.895557 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:01.895111    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerID="a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029" exitCode=1
Sep 08 13:30:01.895557 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:01.895191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerDied","Data":"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"}
Sep 08 13:30:02.902421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:02.902401    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/0.log"
Sep 08 13:30:02.902722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:02.902477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerStarted","Data":"c4f55b0d86218bc758460d838994aa4a87a0ef4cd8ac5a48b501b780babc45f6"}
Sep 08 13:30:02.902835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:02.902816    2580 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" secret="" err="secret \"fail-s3-sa-f54a2b-dockercfg-6kqb7\" not found"
Sep 08 13:30:02.960149 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:02.960118    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:30:02.995718 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:02.995685    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:02.995871 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:02.995771    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:03.495748773 +0000 UTC m=+799.446402704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:03.500157 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:03.500122    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:03.500316 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:03.500210    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:04.500190042 +0000 UTC m=+800.450843974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:03.908745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:03.908666    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerID="7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2" exitCode=0
Sep 08 13:30:03.909103 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:03.908742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerDied","Data":"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"}
Sep 08 13:30:03.909103 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:03.908901    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer" containerID="cri-o://c4f55b0d86218bc758460d838994aa4a87a0ef4cd8ac5a48b501b780babc45f6" gracePeriod=30
Sep 08 13:30:04.508813 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:04.508783    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:04.508968 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:04.508851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:06.508836023 +0000 UTC m=+802.459489953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:06.532193 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:06.532129    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:06.532614 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:06.532229    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:10.532209785 +0000 UTC m=+806.482863722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:08.170464 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.169514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:30:08.175425 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.174992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.177829 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.177804    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 08 13:30:08.177940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.177825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 08 13:30:08.196575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.196548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:30:08.248825 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.248742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.248991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.248886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rtk\" (UniqueName: \"kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.249064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.248996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.249064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.249044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.249190 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.249084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.349929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.350024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rtk\" (UniqueName: \"kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.350096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.350130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.350179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351467 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.351095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.351495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.351940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.351912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.354661 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.354637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.361948 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.361902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rtk\" (UniqueName: \"kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hg2mg\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.499072 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.498963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:08.948369 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.948330    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/1.log"
Sep 08 13:30:08.948847 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.948814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/0.log"
Sep 08 13:30:08.948992 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.948856    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerID="c4f55b0d86218bc758460d838994aa4a87a0ef4cd8ac5a48b501b780babc45f6" exitCode=1
Sep 08 13:30:08.948992 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.948975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerDied","Data":"c4f55b0d86218bc758460d838994aa4a87a0ef4cd8ac5a48b501b780babc45f6"}
Sep 08 13:30:08.949115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:08.949043    2580 scope.go:117] "RemoveContainer" containerID="a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"
Sep 08 13:30:10.572829 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:10.572797    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-serving-cert: secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:10.573267 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:10.572871    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls podName:8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:18.572848982 +0000 UTC m=+814.523502917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls") pod "isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8") : secret "isvc-init-fail-f54a2b-predictor-serving-cert" not found
Sep 08 13:30:12.494338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.494317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/1.log"
Sep 08 13:30:12.496234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.496209    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:30:12.591362 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") pod \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") "
Sep 08 13:30:12.591521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config\") pod \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") "
Sep 08 13:30:12.591521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd859\" (UniqueName: \"kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859\") pod \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") "
Sep 08 13:30:12.591521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591472    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert\") pod \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") "
Sep 08 13:30:12.591521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location\") pod \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\" (UID: \"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8\") "
Sep 08 13:30:12.591802 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:30:12.591881 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:12.591932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.591860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:12.594494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.594458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859" (OuterVolumeSpecName: "kube-api-access-cd859") pod "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8"). InnerVolumeSpecName "kube-api-access-cd859". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:30:12.594724 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.594698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" (UID: "8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:30:12.692291 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.692249    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:30:12.692291 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.692288    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:30:12.692541 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.692305    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd859\" (UniqueName: \"kubernetes.io/projected/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kube-api-access-cd859\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:30:12.692541 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.692318    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:30:12.692541 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.692327    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:30:12.973256 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.973227    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9_8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/storage-initializer/1.log"
Sep 08 13:30:12.973697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.973668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9" event={"ID":"8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8","Type":"ContainerDied","Data":"1c21dcc9c87f3b638e5dfdd131b271a4ee3df6f835af86f780202985cb67540d"}
Sep 08 13:30:12.973820 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:12.973720    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"
Sep 08 13:30:13.040083 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:13.040041    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:30:13.047478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:13.047445    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f54a2b-predictor-69ddccf864-lccf9"]
Sep 08 13:30:14.537653 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:14.537616    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" path="/var/lib/kubelet/pods/8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8/volumes"
Sep 08 13:30:16.863639 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:16.863424    2580 scope.go:117] "RemoveContainer" containerID="c4f55b0d86218bc758460d838994aa4a87a0ef4cd8ac5a48b501b780babc45f6"
Sep 08 13:30:16.903946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:16.902975    2580 scope.go:117] "RemoveContainer" containerID="a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"
Sep 08 13:30:16.903946 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:30:16.903739    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029\": container with ID starting with a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029 not found: ID does not exist" containerID="a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"
Sep 08 13:30:16.903946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:16.903840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029"} err="failed to get container status \"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029\": rpc error: code = NotFound desc = could not find container \"a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029\": container with ID starting with a067f423aea9157e5d12cfc4715b46ff92076f1f26d9692d86db2e9548586029 not found: ID does not exist"
Sep 08 13:30:17.133518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:17.133447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:30:17.136341 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:30:17.136311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38365074_8f1a_4714_9df9_725d43284685.slice/crio-adf6662d4c5878eec9683cbd6f48f7bb4b261ae6250f12ddb56d17b8b4f51677 WatchSource:0}: Error finding container adf6662d4c5878eec9683cbd6f48f7bb4b261ae6250f12ddb56d17b8b4f51677: Status 404 returned error can't find the container with id adf6662d4c5878eec9683cbd6f48f7bb4b261ae6250f12ddb56d17b8b4f51677
Sep 08 13:30:18.005715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:18.005616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerStarted","Data":"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"}
Sep 08 13:30:18.007654 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:18.007599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerStarted","Data":"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"}
Sep 08 13:30:18.007788 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:18.007661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerStarted","Data":"adf6662d4c5878eec9683cbd6f48f7bb4b261ae6250f12ddb56d17b8b4f51677"}
Sep 08 13:30:20.020469 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:20.020433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerStarted","Data":"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"}
Sep 08 13:30:21.028093 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:21.028058    2580 generic.go:356] "Generic (PLEG): container finished" podID="38365074-8f1a-4714-9df9-725d43284685" containerID="062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795" exitCode=0
Sep 08 13:30:21.028093 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:21.028100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerDied","Data":"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"}
Sep 08 13:30:24.063792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.063730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerStarted","Data":"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"}
Sep 08 13:30:24.064403 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.064257    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:30:24.064403 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.064320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:30:24.064403 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.064357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:30:24.067917 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.067881    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:24.070654 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.070603    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:24.072902 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.072830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:30:24.090458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:24.089632    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podStartSLOduration=3.3890236209999998 podStartE2EDuration="26.089612909s" totalImagesPullingTime="19.115645197s" totalInitContainerRuntime="3.584944091s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:24.088048559 +0000 UTC m=+820.038702524" watchObservedRunningTime="2026-09-08 13:30:24.089612909 +0000 UTC m=+820.040266862"
Sep 08 13:30:25.071302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:25.071249    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:25.071912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:25.071851    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:26.077437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:26.077397    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:26.077845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:26.077732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:36.077821 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:36.077774    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:36.078354 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:36.078202    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:45.189079 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:45.188986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerStarted","Data":"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"}
Sep 08 13:30:45.189577 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:45.189092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerStarted","Data":"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"}
Sep 08 13:30:45.189577 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:45.189337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:45.214024 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:45.213969    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podStartSLOduration=9.958490744 podStartE2EDuration="37.213955175s" totalImagesPullingTime="23.506799856s" totalInitContainerRuntime="3.748664575s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:45.211995938 +0000 UTC m=+841.162649893" watchObservedRunningTime="2026-09-08 13:30:45.213955175 +0000 UTC m=+841.164609126"
Sep 08 13:30:46.077870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:46.077826    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:46.078370 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:46.078343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:46.194433 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:46.194394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:46.195743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:46.195712    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:30:47.199379 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:47.199334    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:30:52.203654 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:52.203621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:30:52.204147 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:52.204120    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:30:56.078052 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:56.078008    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:30:56.078626 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:30:56.078499    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:02.204521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:02.204476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:06.077984 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:06.077940    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:31:06.078417 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:06.078385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:12.204475 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:12.204431    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:16.078220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:16.078158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:31:16.078697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:16.078661    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:22.204137 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:22.204094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:26.077571 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:26.077521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:31:26.078032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:26.078000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:32.204356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:32.204317    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:36.077899 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:36.077846    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:31:36.078374 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:36.078348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:42.204635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:42.204598    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:44.490460 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:44.490435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:31:44.491392 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:44.491367    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:31:46.084104 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:46.078637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:31:46.084104 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:46.078827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:31:52.204569 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:52.204529    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:31:53.044196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.041970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:31:53.044196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.042983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" containerID="cri-o://b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2" gracePeriod=30
Sep 08 13:31:53.044196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.043343    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" containerID="cri-o://33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7" gracePeriod=30
Sep 08 13:31:53.044196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.043515    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" containerID="cri-o://98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a" gracePeriod=30
Sep 08 13:31:53.119055 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:31:53.119443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.119535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119444    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.119535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.119535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119458    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.119535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119521    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.119665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.119631    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9bad24-c0d9-4e1d-bda5-32c56a79b9e8" containerName="storage-initializer"
Sep 08 13:31:53.121884 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.121867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.124483 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.124452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 08 13:31:53.124483 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.124472    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 08 13:31:53.133401 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.133379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:31:53.250432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.250399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22g7\" (UniqueName: \"kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.250432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.250442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.250844 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.250536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.250844 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.250615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.250844 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.250660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.351922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.351983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22g7\" (UniqueName: \"kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352544 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.352733 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.352676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.354624 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.354597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.361425 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.361393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22g7\" (UniqueName: \"kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7\") pod \"isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.432506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.432470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:53.520511 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.520480    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerID="33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7" exitCode=2
Sep 08 13:31:53.520665 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.520558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerDied","Data":"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"}
Sep 08 13:31:53.710446 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:53.710417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:31:53.713305 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:31:53.713278    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ab0e3f_3222_4ebb_8027_14e7f781fa7e.slice/crio-82fa2a6e4bf407e56bdd89d1edddaa445138e0b547c7ca437e9e4d62690a8e2c WatchSource:0}: Error finding container 82fa2a6e4bf407e56bdd89d1edddaa445138e0b547c7ca437e9e4d62690a8e2c: Status 404 returned error can't find the container with id 82fa2a6e4bf407e56bdd89d1edddaa445138e0b547c7ca437e9e4d62690a8e2c
Sep 08 13:31:54.064657 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:54.064565    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:31:54.527474 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:54.527435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerStarted","Data":"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"}
Sep 08 13:31:54.527922 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:54.527518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerStarted","Data":"82fa2a6e4bf407e56bdd89d1edddaa445138e0b547c7ca437e9e4d62690a8e2c"}
Sep 08 13:31:56.078141 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:56.078082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:31:56.078531 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:56.078447    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:57.544877 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:57.544841    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerID="cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942" exitCode=0
Sep 08 13:31:57.544877 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:57.544880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerDied","Data":"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"}
Sep 08 13:31:58.551712 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.551680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerStarted","Data":"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"}
Sep 08 13:31:58.552142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.551749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerStarted","Data":"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"}
Sep 08 13:31:58.552142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.551782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerStarted","Data":"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"}
Sep 08 13:31:58.552283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.552139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:58.552283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.552218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:58.552283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.552250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:31:58.553694 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.553658    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:31:58.554377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.554344    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:58.554463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.554392    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerID="b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2" exitCode=0
Sep 08 13:31:58.554463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.554424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerDied","Data":"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"}
Sep 08 13:31:58.577947 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:58.577894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podStartSLOduration=1.9762022780000001 podStartE2EDuration="5.577877754s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.601675476s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:58.576940036 +0000 UTC m=+914.527593987" watchObservedRunningTime="2026-09-08 13:31:58.577877754 +0000 UTC m=+914.528531705"
Sep 08 13:31:59.064611 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:59.064566    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:31:59.559793 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:59.559743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:31:59.560232 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:31:59.560204    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:02.205530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:02.205500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:32:04.064870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:04.064829    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:32:04.065331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:04.064948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:32:04.563704 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:04.563674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:32:04.564366 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:04.564336    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:04.564576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:04.564552    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:06.077895 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:06.077847    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:32:06.078340 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:06.078150    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:08.221373 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:08.221343    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:32:08.221789 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:08.221576    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" containerID="cri-o://f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96" gracePeriod=30
Sep 08 13:32:08.221789 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:08.221622    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kube-rbac-proxy" containerID="cri-o://5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7" gracePeriod=30
Sep 08 13:32:08.604220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:08.604118    2580 generic.go:356] "Generic (PLEG): container finished" podID="38365074-8f1a-4714-9df9-725d43284685" containerID="5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7" exitCode=2
Sep 08 13:32:08.604373 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:08.604194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerDied","Data":"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"}
Sep 08 13:32:09.064565 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:09.064528    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:32:12.200181 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:12.200119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 08 13:32:12.204845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:12.204816    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 08 13:32:12.969959 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:12.969934    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:32:13.130025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.129939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls\") pod \"38365074-8f1a-4714-9df9-725d43284685\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") "
Sep 08 13:32:13.130025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.129982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config\") pod \"38365074-8f1a-4714-9df9-725d43284685\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") "
Sep 08 13:32:13.130025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8rtk\" (UniqueName: \"kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk\") pod \"38365074-8f1a-4714-9df9-725d43284685\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") "
Sep 08 13:32:13.130347 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert\") pod \"38365074-8f1a-4714-9df9-725d43284685\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") "
Sep 08 13:32:13.130347 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location\") pod \"38365074-8f1a-4714-9df9-725d43284685\" (UID: \"38365074-8f1a-4714-9df9-725d43284685\") "
Sep 08 13:32:13.130463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130430    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38365074-8f1a-4714-9df9-725d43284685" (UID: "38365074-8f1a-4714-9df9-725d43284685"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:13.130519 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130463    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38365074-8f1a-4714-9df9-725d43284685" (UID: "38365074-8f1a-4714-9df9-725d43284685"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:32:13.130519 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.130476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38365074-8f1a-4714-9df9-725d43284685" (UID: "38365074-8f1a-4714-9df9-725d43284685"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:13.132099 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.132069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38365074-8f1a-4714-9df9-725d43284685" (UID: "38365074-8f1a-4714-9df9-725d43284685"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:32:13.132302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.132127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk" (OuterVolumeSpecName: "kube-api-access-c8rtk") pod "38365074-8f1a-4714-9df9-725d43284685" (UID: "38365074-8f1a-4714-9df9-725d43284685"). InnerVolumeSpecName "kube-api-access-c8rtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:32:13.231434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.231403    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:13.231434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.231432    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rtk\" (UniqueName: \"kubernetes.io/projected/38365074-8f1a-4714-9df9-725d43284685-kube-api-access-c8rtk\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:13.231434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.231443    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38365074-8f1a-4714-9df9-725d43284685-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:13.231856 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.231452    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38365074-8f1a-4714-9df9-725d43284685-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:13.231856 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.231463    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38365074-8f1a-4714-9df9-725d43284685-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:13.433683 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.433583    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:32:13.433974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.433962    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kube-rbac-proxy"
Sep 08 13:32:13.434023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.433976    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kube-rbac-proxy"
Sep 08 13:32:13.434023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.433995    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container"
Sep 08 13:32:13.434023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.434001    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container"
Sep 08 13:32:13.434023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.434018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="storage-initializer"
Sep 08 13:32:13.434023 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.434024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="storage-initializer"
Sep 08 13:32:13.434185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.434079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kube-rbac-proxy"
Sep 08 13:32:13.434185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.434091    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38365074-8f1a-4714-9df9-725d43284685" containerName="kserve-container"
Sep 08 13:32:13.437668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.437648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.440223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.440181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 08 13:32:13.440325 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.440309    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 08 13:32:13.449078 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.449055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:32:13.630701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.630666    2580 generic.go:356] "Generic (PLEG): container finished" podID="38365074-8f1a-4714-9df9-725d43284685" containerID="f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96" exitCode=0
Sep 08 13:32:13.630835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.630733    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"
Sep 08 13:32:13.630835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.630739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerDied","Data":"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"}
Sep 08 13:32:13.630835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.630800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg" event={"ID":"38365074-8f1a-4714-9df9-725d43284685","Type":"ContainerDied","Data":"adf6662d4c5878eec9683cbd6f48f7bb4b261ae6250f12ddb56d17b8b4f51677"}
Sep 08 13:32:13.630953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.630841    2580 scope.go:117] "RemoveContainer" containerID="5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"
Sep 08 13:32:13.634093 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.634064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.634257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.634109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.634257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.634181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.634356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.634266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jcd\" (UniqueName: \"kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.634356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.634306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.640094 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.640076    2580 scope.go:117] "RemoveContainer" containerID="f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"
Sep 08 13:32:13.647588 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.647574    2580 scope.go:117] "RemoveContainer" containerID="062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"
Sep 08 13:32:13.655021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655000    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:32:13.655305 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655292    2580 scope.go:117] "RemoveContainer" containerID="5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"
Sep 08 13:32:13.655572 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:13.655553    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7\": container with ID starting with 5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7 not found: ID does not exist" containerID="5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"
Sep 08 13:32:13.655625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655583    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7"} err="failed to get container status \"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7\": rpc error: code = NotFound desc = could not find container \"5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7\": container with ID starting with 5a139f180df75cd8580ccad8465e59d9b3b11d88476070c3c8f38639b8e040b7 not found: ID does not exist"
Sep 08 13:32:13.655625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655602    2580 scope.go:117] "RemoveContainer" containerID="f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"
Sep 08 13:32:13.655875 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:13.655857    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96\": container with ID starting with f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96 not found: ID does not exist" containerID="f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"
Sep 08 13:32:13.655929 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655883    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96"} err="failed to get container status \"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96\": rpc error: code = NotFound desc = could not find container \"f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96\": container with ID starting with f8c2ad58296fe16dde3a88cbdee122447bc101b1df474c4fc2669f41a3c91b96 not found: ID does not exist"
Sep 08 13:32:13.655929 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.655899    2580 scope.go:117] "RemoveContainer" containerID="062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"
Sep 08 13:32:13.656128 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:13.656114    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795\": container with ID starting with 062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795 not found: ID does not exist" containerID="062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"
Sep 08 13:32:13.656189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.656133    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795"} err="failed to get container status \"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795\": rpc error: code = NotFound desc = could not find container \"062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795\": container with ID starting with 062c6efd896f097309b4cb8177b62d1c53aef010961df2d5aa7df7e04cd42795 not found: ID does not exist"
Sep 08 13:32:13.660544 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.660521    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hg2mg"]
Sep 08 13:32:13.734944 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.734919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.734952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.734982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.735023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jcd\" (UniqueName: \"kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.735046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.735698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.735740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.735815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.735766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.737592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.737569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.744012 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.743992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jcd\" (UniqueName: \"kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd\") pod \"isvc-predictive-xgboost-predictor-695f978978-mxcrd\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:13.748211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:13.748190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:14.023769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.023742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:32:14.025972 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:32:14.025946    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2272de_f3e1_4ede_8bf0_a12180d63612.slice/crio-b2ff9ec20c9fac2de299b6a866a12c8761ca62f8ee9b8e6b8f54dbe0ccdbf328 WatchSource:0}: Error finding container b2ff9ec20c9fac2de299b6a866a12c8761ca62f8ee9b8e6b8f54dbe0ccdbf328: Status 404 returned error can't find the container with id b2ff9ec20c9fac2de299b6a866a12c8761ca62f8ee9b8e6b8f54dbe0ccdbf328
Sep 08 13:32:14.065060 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.065029    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:32:14.536488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.536454    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38365074-8f1a-4714-9df9-725d43284685" path="/var/lib/kubelet/pods/38365074-8f1a-4714-9df9-725d43284685/volumes"
Sep 08 13:32:14.564730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.564682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:14.565106 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.565075    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:14.639006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.638970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerStarted","Data":"e3ff95facdcf82cb89a7783dfa2be87252b75ab050d390df4053eac579a7e4ac"}
Sep 08 13:32:14.639200 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:14.639025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerStarted","Data":"b2ff9ec20c9fac2de299b6a866a12c8761ca62f8ee9b8e6b8f54dbe0ccdbf328"}
Sep 08 13:32:16.077692 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:16.077638    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 08 13:32:16.078074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:16.077785    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:32:16.078074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:16.077868    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:16.078074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:16.077960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:32:18.660046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:18.660014    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerID="e3ff95facdcf82cb89a7783dfa2be87252b75ab050d390df4053eac579a7e4ac" exitCode=0
Sep 08 13:32:18.660540 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:18.660090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerDied","Data":"e3ff95facdcf82cb89a7783dfa2be87252b75ab050d390df4053eac579a7e4ac"}
Sep 08 13:32:19.064696 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:19.064649    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 08 13:32:19.667185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:19.667128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerStarted","Data":"cc3e0195a3071b2f7d63457085e53442267945a78fe62d72bf4a1c19d056f971"}
Sep 08 13:32:19.667681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:19.667239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerStarted","Data":"c9e2471f435f67b175fb35b7cd826f49c2daee0b219863c15fd2c9e94941a80f"}
Sep 08 13:32:19.667681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:19.667464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:19.691124 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:19.691072    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podStartSLOduration=3.129675874 podStartE2EDuration="6.691056945s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.561381071s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:19.688907652 +0000 UTC m=+935.639561606" watchObservedRunningTime="2026-09-08 13:32:19.691056945 +0000 UTC m=+935.641710897"
Sep 08 13:32:20.674668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:20.674633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:20.676018 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:20.675983    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:32:21.679135 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:21.679096    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:32:23.197621 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.197599    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:32:23.204119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location\") pod \"dc3b8fad-2b03-457f-85bf-564719775fe5\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") "
Sep 08 13:32:23.204293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") pod \"dc3b8fad-2b03-457f-85bf-564719775fe5\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") "
Sep 08 13:32:23.204293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert\") pod \"dc3b8fad-2b03-457f-85bf-564719775fe5\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") "
Sep 08 13:32:23.204293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwlzm\" (UniqueName: \"kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm\") pod \"dc3b8fad-2b03-457f-85bf-564719775fe5\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") "
Sep 08 13:32:23.204513 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc3b8fad-2b03-457f-85bf-564719775fe5" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:32:23.204608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.204581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc3b8fad-2b03-457f-85bf-564719775fe5" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:23.206260 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.206231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc3b8fad-2b03-457f-85bf-564719775fe5" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:32:23.206439 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.206418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm" (OuterVolumeSpecName: "kube-api-access-cwlzm") pod "dc3b8fad-2b03-457f-85bf-564719775fe5" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5"). InnerVolumeSpecName "kube-api-access-cwlzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:32:23.305504 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config\") pod \"dc3b8fad-2b03-457f-85bf-564719775fe5\" (UID: \"dc3b8fad-2b03-457f-85bf-564719775fe5\") "
Sep 08 13:32:23.305652 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305575    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc3b8fad-2b03-457f-85bf-564719775fe5-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:23.305652 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305587    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc3b8fad-2b03-457f-85bf-564719775fe5-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:23.305652 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305596    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:23.305652 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305605    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlzm\" (UniqueName: \"kubernetes.io/projected/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-api-access-cwlzm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:23.305778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.305736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc3b8fad-2b03-457f-85bf-564719775fe5" (UID: "dc3b8fad-2b03-457f-85bf-564719775fe5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:23.405981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.405947    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc3b8fad-2b03-457f-85bf-564719775fe5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:32:23.690506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.690414    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerID="98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a" exitCode=0
Sep 08 13:32:23.690506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.690479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"
Sep 08 13:32:23.690506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.690485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerDied","Data":"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"}
Sep 08 13:32:23.690738 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.690551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4" event={"ID":"dc3b8fad-2b03-457f-85bf-564719775fe5","Type":"ContainerDied","Data":"7dc92dc8db426279b66022c18f5bd7c489ca7302f4f0de10e6d9780912f93e26"}
Sep 08 13:32:23.690738 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.690583    2580 scope.go:117] "RemoveContainer" containerID="98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"
Sep 08 13:32:23.699377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.699361    2580 scope.go:117] "RemoveContainer" containerID="33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"
Sep 08 13:32:23.706914 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.706897    2580 scope.go:117] "RemoveContainer" containerID="b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"
Sep 08 13:32:23.714103 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.714085    2580 scope.go:117] "RemoveContainer" containerID="7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"
Sep 08 13:32:23.718333 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.718311    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:32:23.721767 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.721751    2580 scope.go:117] "RemoveContainer" containerID="98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"
Sep 08 13:32:23.722025 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:23.722004    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a\": container with ID starting with 98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a not found: ID does not exist" containerID="98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"
Sep 08 13:32:23.722091 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722035    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a"} err="failed to get container status \"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a\": rpc error: code = NotFound desc = could not find container \"98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a\": container with ID starting with 98fb33fb103a79a3171fd0f6fdf5ecfa44baa90fdf2a55f8578a0c5c2d59f81a not found: ID does not exist"
Sep 08 13:32:23.722091 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722053    2580 scope.go:117] "RemoveContainer" containerID="33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"
Sep 08 13:32:23.722355 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:23.722331    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7\": container with ID starting with 33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7 not found: ID does not exist" containerID="33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"
Sep 08 13:32:23.722450 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722358    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7"} err="failed to get container status \"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7\": rpc error: code = NotFound desc = could not find container \"33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7\": container with ID starting with 33bb1b95826cde7b6766ee22a32a238a72ba7556151b442857fca4ef401a6db7 not found: ID does not exist"
Sep 08 13:32:23.722450 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722376    2580 scope.go:117] "RemoveContainer" containerID="b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"
Sep 08 13:32:23.722642 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:23.722623    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2\": container with ID starting with b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2 not found: ID does not exist" containerID="b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"
Sep 08 13:32:23.722712 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722651    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2"} err="failed to get container status \"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2\": rpc error: code = NotFound desc = could not find container \"b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2\": container with ID starting with b8827a95fd07cb71bf9194a29d7525d8bae6b06aefd4f6c107edc4dfab8ddbd2 not found: ID does not exist"
Sep 08 13:32:23.722712 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.722672    2580 scope.go:117] "RemoveContainer" containerID="7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"
Sep 08 13:32:23.725512 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:32:23.725265    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2\": container with ID starting with 7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2 not found: ID does not exist" containerID="7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"
Sep 08 13:32:23.725512 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.725305    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2"} err="failed to get container status \"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2\": rpc error: code = NotFound desc = could not find container \"7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2\": container with ID starting with 7bd4c1f6dcb95d30ce52affbe7bae24c641ea4bc1ed3586f0de9f61914de78f2 not found: ID does not exist"
Sep 08 13:32:23.726095 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:23.726079    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5bc7c474cb-kctr4"]
Sep 08 13:32:24.535582 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:24.535545    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" path="/var/lib/kubelet/pods/dc3b8fad-2b03-457f-85bf-564719775fe5/volumes"
Sep 08 13:32:24.564833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:24.564803    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:24.565241 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:24.565216    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:26.684161 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:26.684130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:32:26.684808 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:26.684777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:32:34.564467 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:34.564426    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:34.565300 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:34.565277    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:36.684849 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:36.684810    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:32:44.564397 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:44.564354    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:44.564833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:44.564808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:46.685309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:46.685266    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:32:54.564868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:54.564827    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:32:54.565315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:54.565291    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:32:56.685069 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:32:56.685027    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:33:04.564343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:04.564305    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:33:04.564872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:04.564544    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:06.685308 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:06.685261    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:33:14.565357 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:14.565316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:14.565866 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:14.565845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:16.685145 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:16.685105    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:33:26.685672 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:26.685634    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:33:28.168894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.168863    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:33:28.169339 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.169107    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" containerID="cri-o://34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2" gracePeriod=30
Sep 08 13:33:28.169339 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.169140    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" containerID="cri-o://4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9" gracePeriod=30
Sep 08 13:33:28.169339 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.169206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" containerID="cri-o://7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d" gracePeriod=30
Sep 08 13:33:28.268215 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268178    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:33:28.268638 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268621    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268640    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268670    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="storage-initializer"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268679    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="storage-initializer"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268727    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy"
Sep 08 13:33:28.268740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy"
Sep 08 13:33:28.269084 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268827    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kube-rbac-proxy"
Sep 08 13:33:28.269084 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="agent"
Sep 08 13:33:28.269084 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.268851    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3b8fad-2b03-457f-85bf-564719775fe5" containerName="kserve-container"
Sep 08 13:33:28.272290 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.272268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.274936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.274909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 08 13:33:28.275036 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.274933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 08 13:33:28.281646 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.281623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:33:28.355205 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.355147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.355352 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.355228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.355352 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.355261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mns2v\" (UniqueName: \"kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.456380 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.456348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mns2v\" (UniqueName: \"kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.456555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.456425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.456555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.456451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.457067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.457045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.458988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.458960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.465299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.465268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mns2v\" (UniqueName: \"kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v\") pod \"message-dumper-predictor-6575bcfdfd-2mc9n\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.584757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.584718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:28.861311 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:28.861282    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:33:28.863249 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:33:28.863223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87b85ed_af72_4355_8568_9baad12814c4.slice/crio-ee01bb03f2562b63eacb76ffd2d64a9231a97bd81cad85c232d9c8a6c5946168 WatchSource:0}: Error finding container ee01bb03f2562b63eacb76ffd2d64a9231a97bd81cad85c232d9c8a6c5946168: Status 404 returned error can't find the container with id ee01bb03f2562b63eacb76ffd2d64a9231a97bd81cad85c232d9c8a6c5946168
Sep 08 13:33:29.006958 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:29.006923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerStarted","Data":"ee01bb03f2562b63eacb76ffd2d64a9231a97bd81cad85c232d9c8a6c5946168"}
Sep 08 13:33:29.009287 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:29.009261    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerID="7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d" exitCode=2
Sep 08 13:33:29.009405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:29.009294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerDied","Data":"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"}
Sep 08 13:33:29.560341 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:29.560300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:31.021873 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:31.021834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerStarted","Data":"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"}
Sep 08 13:33:31.022316 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:31.021905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerStarted","Data":"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"}
Sep 08 13:33:31.022316 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:31.021987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:31.043872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:31.043830    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" podStartSLOduration=1.872471778 podStartE2EDuration="3.043816263s" totalImagesPullingTime="1.171344485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:31.041053553 +0000 UTC m=+1006.991707528" watchObservedRunningTime="2026-09-08 13:33:31.043816263 +0000 UTC m=+1006.994470265"
Sep 08 13:33:32.026553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:32.026515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:32.028547 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:32.028518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:33.033298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:33.033265    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerID="34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2" exitCode=0
Sep 08 13:33:33.033688 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:33.033332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerDied","Data":"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"}
Sep 08 13:33:34.559858 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:34.559820    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:34.565181 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:34.565141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:33:34.566734 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:34.566706    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:36.685494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:36.685460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:33:39.043239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:39.043209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:33:39.560343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:39.560297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:39.560544 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:39.560437    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:43.498338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:43.498308    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:33:43.498835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:43.498520    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" containerID="cri-o://c9e2471f435f67b175fb35b7cd826f49c2daee0b219863c15fd2c9e94941a80f" gracePeriod=30
Sep 08 13:33:43.498835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:43.498555    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kube-rbac-proxy" containerID="cri-o://cc3e0195a3071b2f7d63457085e53442267945a78fe62d72bf4a1c19d056f971" gracePeriod=30
Sep 08 13:33:44.093755 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:44.093711    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerID="cc3e0195a3071b2f7d63457085e53442267945a78fe62d72bf4a1c19d056f971" exitCode=2
Sep 08 13:33:44.093950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:44.093786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerDied","Data":"cc3e0195a3071b2f7d63457085e53442267945a78fe62d72bf4a1c19d056f971"}
Sep 08 13:33:44.560782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:44.560741    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:44.565178 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:44.565141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:33:44.566727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:44.566702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:46.679363 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:46.679322    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 08 13:33:46.685727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:46.685702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 08 13:33:48.114936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.114901    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerID="c9e2471f435f67b175fb35b7cd826f49c2daee0b219863c15fd2c9e94941a80f" exitCode=0
Sep 08 13:33:48.115408 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.114973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerDied","Data":"c9e2471f435f67b175fb35b7cd826f49c2daee0b219863c15fd2c9e94941a80f"}
Sep 08 13:33:48.155035 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.155012    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:33:48.237407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location\") pod \"fd2272de-f3e1-4ede-8bf0-a12180d63612\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") "
Sep 08 13:33:48.237407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7jcd\" (UniqueName: \"kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd\") pod \"fd2272de-f3e1-4ede-8bf0-a12180d63612\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") "
Sep 08 13:33:48.237600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config\") pod \"fd2272de-f3e1-4ede-8bf0-a12180d63612\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") "
Sep 08 13:33:48.237600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert\") pod \"fd2272de-f3e1-4ede-8bf0-a12180d63612\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") "
Sep 08 13:33:48.237716 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls\") pod \"fd2272de-f3e1-4ede-8bf0-a12180d63612\" (UID: \"fd2272de-f3e1-4ede-8bf0-a12180d63612\") "
Sep 08 13:33:48.237782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd2272de-f3e1-4ede-8bf0-a12180d63612" (UID: "fd2272de-f3e1-4ede-8bf0-a12180d63612"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:48.237835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd2272de-f3e1-4ede-8bf0-a12180d63612" (UID: "fd2272de-f3e1-4ede-8bf0-a12180d63612"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:48.237835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd2272de-f3e1-4ede-8bf0-a12180d63612" (UID: "fd2272de-f3e1-4ede-8bf0-a12180d63612"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:48.237932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237909    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd2272de-f3e1-4ede-8bf0-a12180d63612-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:48.237986 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237931    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:48.237986 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.237946    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd2272de-f3e1-4ede-8bf0-a12180d63612-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:48.239635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.239595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd" (OuterVolumeSpecName: "kube-api-access-j7jcd") pod "fd2272de-f3e1-4ede-8bf0-a12180d63612" (UID: "fd2272de-f3e1-4ede-8bf0-a12180d63612"). InnerVolumeSpecName "kube-api-access-j7jcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:48.239741 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.239723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd2272de-f3e1-4ede-8bf0-a12180d63612" (UID: "fd2272de-f3e1-4ede-8bf0-a12180d63612"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:48.305037 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305003    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:33:48.305401 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kube-rbac-proxy"
Sep 08 13:33:48.305454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305403    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kube-rbac-proxy"
Sep 08 13:33:48.305454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305420    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container"
Sep 08 13:33:48.305454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305427    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container"
Sep 08 13:33:48.305454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305433    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="storage-initializer"
Sep 08 13:33:48.305454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305439    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="storage-initializer"
Sep 08 13:33:48.305608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305500    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kserve-container"
Sep 08 13:33:48.305608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.305510    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" containerName="kube-rbac-proxy"
Sep 08 13:33:48.308947 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.308930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.311774 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.311738    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 08 13:33:48.311874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.311809    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 08 13:33:48.321627 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.321601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:33:48.338941 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.338913    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd2272de-f3e1-4ede-8bf0-a12180d63612-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:48.338941 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.338938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7jcd\" (UniqueName: \"kubernetes.io/projected/fd2272de-f3e1-4ede-8bf0-a12180d63612-kube-api-access-j7jcd\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:48.439518 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.439479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.439691 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.439535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.439691 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.439598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxphg\" (UniqueName: \"kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.439691 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.439649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.439691 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.439674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxphg\" (UniqueName: \"kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.540742 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:48.540535    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 08 13:33:48.540742 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:48.540608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls podName:bc357685-7683-4c3f-ad85-a55fc7789625 nodeName:}" failed. No retries permitted until 2026-09-08 13:33:49.040585638 +0000 UTC m=+1024.991239583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls") pod "isvc-logger-predictor-5979d78599-hlwv6" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625") : secret "isvc-logger-predictor-serving-cert" not found
Sep 08 13:33:48.541003 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.540873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.541149 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.541129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.541258 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.541238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:48.549708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:48.549687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxphg\" (UniqueName: \"kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:49.046007 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.045972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:49.048446 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.048425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") pod \"isvc-logger-predictor-5979d78599-hlwv6\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:49.121644 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.121616    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"
Sep 08 13:33:49.122018 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.121612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd" event={"ID":"fd2272de-f3e1-4ede-8bf0-a12180d63612","Type":"ContainerDied","Data":"b2ff9ec20c9fac2de299b6a866a12c8761ca62f8ee9b8e6b8f54dbe0ccdbf328"}
Sep 08 13:33:49.122838 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.121775    2580 scope.go:117] "RemoveContainer" containerID="cc3e0195a3071b2f7d63457085e53442267945a78fe62d72bf4a1c19d056f971"
Sep 08 13:33:49.131178 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.131147    2580 scope.go:117] "RemoveContainer" containerID="c9e2471f435f67b175fb35b7cd826f49c2daee0b219863c15fd2c9e94941a80f"
Sep 08 13:33:49.138463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.138444    2580 scope.go:117] "RemoveContainer" containerID="e3ff95facdcf82cb89a7783dfa2be87252b75ab050d390df4053eac579a7e4ac"
Sep 08 13:33:49.144611 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.144586    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:33:49.155056 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.155033    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-mxcrd"]
Sep 08 13:33:49.218927 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.218901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:49.500806 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.500782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:33:49.502565 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:33:49.502538    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc357685_7683_4c3f_ad85_a55fc7789625.slice/crio-61a016d276254903c48362a5962f47777619053ca8a3cfe9c288c7f30e8cda91 WatchSource:0}: Error finding container 61a016d276254903c48362a5962f47777619053ca8a3cfe9c288c7f30e8cda91: Status 404 returned error can't find the container with id 61a016d276254903c48362a5962f47777619053ca8a3cfe9c288c7f30e8cda91
Sep 08 13:33:49.559993 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:49.559960    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:50.129889 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:50.129848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerStarted","Data":"e6572fa48577ea49362ee709640ed12bc72454b32561cccd83d7acea06a54f93"}
Sep 08 13:33:50.130379 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:50.129929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerStarted","Data":"61a016d276254903c48362a5962f47777619053ca8a3cfe9c288c7f30e8cda91"}
Sep 08 13:33:50.536090 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:50.536058    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2272de-f3e1-4ede-8bf0-a12180d63612" path="/var/lib/kubelet/pods/fd2272de-f3e1-4ede-8bf0-a12180d63612/volumes"
Sep 08 13:33:54.150657 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.150624    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc357685-7683-4c3f-ad85-a55fc7789625" containerID="e6572fa48577ea49362ee709640ed12bc72454b32561cccd83d7acea06a54f93" exitCode=0
Sep 08 13:33:54.151040 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.150699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerDied","Data":"e6572fa48577ea49362ee709640ed12bc72454b32561cccd83d7acea06a54f93"}
Sep 08 13:33:54.560264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.560226    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 08 13:33:54.564471 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.564440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:5000: connect: connection refused"
Sep 08 13:33:54.564589 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.564561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:54.567146 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.567113    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:54.567299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:54.567282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:55.158116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.158079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerStarted","Data":"a93ac01050a85b11a8d2aeab5c2a44895c7a975bc17d4ed2c49f09ee930953b5"}
Sep 08 13:33:55.158485 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.158146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerStarted","Data":"3cccd85b1da6acefd91bc704b13e77d90b74d72758a5db98c2700c928d4bcd66"}
Sep 08 13:33:55.158485 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.158183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerStarted","Data":"12d7a50da7aea34f4ce704281f0fc1eca1d79cd8751c1c9ef6c916136cac7334"}
Sep 08 13:33:55.158485 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.158440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:55.158668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.158621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:55.159859 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.159835    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:33:55.183758 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:55.183719    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podStartSLOduration=3.249764715 podStartE2EDuration="7.183705335s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.93394062s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:55.182453192 +0000 UTC m=+1031.133107146" watchObservedRunningTime="2026-09-08 13:33:55.183705335 +0000 UTC m=+1031.134359346"
Sep 08 13:33:56.163704 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:56.163666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:33:56.164093 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:56.163795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:33:56.164848 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:56.164819    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:57.169066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:57.169020    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:33:57.169545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:57.169452    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:33:58.310063 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.310036    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:58.325046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q22g7\" (UniqueName: \"kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7\") pod \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") "
Sep 08 13:33:58.325214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls\") pod \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") "
Sep 08 13:33:58.325214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config\") pod \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") "
Sep 08 13:33:58.325342 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert\") pod \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") "
Sep 08 13:33:58.325342 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location\") pod \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\" (UID: \"03ab0e3f-3222-4ebb-8027-14e7f781fa7e\") "
Sep 08 13:33:58.325614 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03ab0e3f-3222-4ebb-8027-14e7f781fa7e" (UID: "03ab0e3f-3222-4ebb-8027-14e7f781fa7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:58.325715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03ab0e3f-3222-4ebb-8027-14e7f781fa7e" (UID: "03ab0e3f-3222-4ebb-8027-14e7f781fa7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:58.325715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.325590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03ab0e3f-3222-4ebb-8027-14e7f781fa7e" (UID: "03ab0e3f-3222-4ebb-8027-14e7f781fa7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:58.327096 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.327078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7" (OuterVolumeSpecName: "kube-api-access-q22g7") pod "03ab0e3f-3222-4ebb-8027-14e7f781fa7e" (UID: "03ab0e3f-3222-4ebb-8027-14e7f781fa7e"). InnerVolumeSpecName "kube-api-access-q22g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:58.327240 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.327218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03ab0e3f-3222-4ebb-8027-14e7f781fa7e" (UID: "03ab0e3f-3222-4ebb-8027-14e7f781fa7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:58.426434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.426354    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:58.426434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.426382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:58.426434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.426392    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:58.426434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.426402    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:58.426434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:58.426410    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q22g7\" (UniqueName: \"kubernetes.io/projected/03ab0e3f-3222-4ebb-8027-14e7f781fa7e-kube-api-access-q22g7\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:33:59.180478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.180443    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerID="4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9" exitCode=0
Sep 08 13:33:59.180651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.180505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerDied","Data":"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"}
Sep 08 13:33:59.180651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.180524    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"
Sep 08 13:33:59.180651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.180561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5" event={"ID":"03ab0e3f-3222-4ebb-8027-14e7f781fa7e","Type":"ContainerDied","Data":"82fa2a6e4bf407e56bdd89d1edddaa445138e0b547c7ca437e9e4d62690a8e2c"}
Sep 08 13:33:59.180651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.180591    2580 scope.go:117] "RemoveContainer" containerID="4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"
Sep 08 13:33:59.189033 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.189013    2580 scope.go:117] "RemoveContainer" containerID="7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"
Sep 08 13:33:59.196610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.196591    2580 scope.go:117] "RemoveContainer" containerID="34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"
Sep 08 13:33:59.200972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.200952    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:33:59.204570 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.204553    2580 scope.go:117] "RemoveContainer" containerID="cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"
Sep 08 13:33:59.204971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.204948    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7956cfc759-sk8j5"]
Sep 08 13:33:59.212028 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212011    2580 scope.go:117] "RemoveContainer" containerID="4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"
Sep 08 13:33:59.212300 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:59.212283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9\": container with ID starting with 4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9 not found: ID does not exist" containerID="4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"
Sep 08 13:33:59.212348 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212308    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9"} err="failed to get container status \"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9\": rpc error: code = NotFound desc = could not find container \"4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9\": container with ID starting with 4dc7e5c2cb62165f2b4ce7c018fcf4452f41f3e3c15a249e5b2ff56916073fc9 not found: ID does not exist"
Sep 08 13:33:59.212348 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212325    2580 scope.go:117] "RemoveContainer" containerID="7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"
Sep 08 13:33:59.212565 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:59.212551    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d\": container with ID starting with 7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d not found: ID does not exist" containerID="7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"
Sep 08 13:33:59.212610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212568    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d"} err="failed to get container status \"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d\": rpc error: code = NotFound desc = could not find container \"7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d\": container with ID starting with 7d174287902d00949c0a349fedac01495e193c391c56a6cdcc06f757f56b471d not found: ID does not exist"
Sep 08 13:33:59.212610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212580    2580 scope.go:117] "RemoveContainer" containerID="34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"
Sep 08 13:33:59.212809 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:59.212791    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2\": container with ID starting with 34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2 not found: ID does not exist" containerID="34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"
Sep 08 13:33:59.212872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212818    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2"} err="failed to get container status \"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2\": rpc error: code = NotFound desc = could not find container \"34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2\": container with ID starting with 34c33ed4d6f23ae632894eb9cd9df5dcf77b4db7229225c949a5280a4b458eb2 not found: ID does not exist"
Sep 08 13:33:59.212872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.212843    2580 scope.go:117] "RemoveContainer" containerID="cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"
Sep 08 13:33:59.213067 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:33:59.213052    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942\": container with ID starting with cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942 not found: ID does not exist" containerID="cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"
Sep 08 13:33:59.213107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:33:59.213071    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942"} err="failed to get container status \"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942\": rpc error: code = NotFound desc = could not find container \"cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942\": container with ID starting with cdf45b9a5e5fa59991973bd44fe745597982852d46a2b4e5f8c9995465884942 not found: ID does not exist"
Sep 08 13:34:00.535597 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:00.535563    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" path="/var/lib/kubelet/pods/03ab0e3f-3222-4ebb-8027-14e7f781fa7e/volumes"
Sep 08 13:34:02.174530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:02.174500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:34:02.175099 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:02.175064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:02.175447 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:02.175422    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:12.175740 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:12.175697    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:12.176232 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:12.176207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:22.175925 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:22.175882    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:22.176434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:22.176352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:32.175401 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:32.175358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:32.176014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:32.175984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:42.175478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:42.175380    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:42.175870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:42.175773    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:52.175077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:52.175038    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:34:52.175534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:34:52.175513    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:02.175000 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:02.174953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:35:02.175487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:02.175434    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:12.175669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:12.175634    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:12.176108 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:12.175897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:23.341826 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.341786    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-2mc9n_f87b85ed-af72-4355-8568-9baad12814c4/kserve-container/0.log"
Sep 08 13:35:23.559450 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.559419    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:35:23.559730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.559679    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" containerID="cri-o://12d7a50da7aea34f4ce704281f0fc1eca1d79cd8751c1c9ef6c916136cac7334" gracePeriod=30
Sep 08 13:35:23.559730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.559715    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" containerID="cri-o://a93ac01050a85b11a8d2aeab5c2a44895c7a975bc17d4ed2c49f09ee930953b5" gracePeriod=30
Sep 08 13:35:23.559907 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.559716    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" containerID="cri-o://3cccd85b1da6acefd91bc704b13e77d90b74d72758a5db98c2700c928d4bcd66" gracePeriod=30
Sep 08 13:35:23.711712 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.711683    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:35:23.711953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.711899    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kserve-container" containerID="cri-o://1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca" gracePeriod=30
Sep 08 13:35:23.711953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.711927    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kube-rbac-proxy" containerID="cri-o://b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6" gracePeriod=30
Sep 08 13:35:23.733392 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733369    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:35:23.733720 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733708    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy"
Sep 08 13:35:23.733764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy"
Sep 08 13:35:23.733764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733743    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="storage-initializer"
Sep 08 13:35:23.733764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733749    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="storage-initializer"
Sep 08 13:35:23.733764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733760    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733766    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733838    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kserve-container"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733845    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="kube-rbac-proxy"
Sep 08 13:35:23.733887 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.733854    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ab0e3f-3222-4ebb-8027-14e7f781fa7e" containerName="agent"
Sep 08 13:35:23.740030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.740009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.742554 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.742534    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 08 13:35:23.742635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.742621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 08 13:35:23.752107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.752086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:35:23.840158 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.840130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhw5r\" (UniqueName: \"kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.840306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.840218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.840306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.840244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.840306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.840272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.840306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.840296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhw5r\" (UniqueName: \"kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941496 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941496 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.941802 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.941775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.942246 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.942048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.942246 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.942156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.943655 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.943633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.951600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.951573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhw5r\" (UniqueName: \"kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r\") pod \"isvc-autogluon-v1-predictor-574b5b7495-vgvg5\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:23.959581 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:23.959563    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:35:24.042203 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.042151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config\") pod \"f87b85ed-af72-4355-8568-9baad12814c4\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") "
Sep 08 13:35:24.042351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.042221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls\") pod \"f87b85ed-af72-4355-8568-9baad12814c4\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") "
Sep 08 13:35:24.042351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.042269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mns2v\" (UniqueName: \"kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v\") pod \"f87b85ed-af72-4355-8568-9baad12814c4\" (UID: \"f87b85ed-af72-4355-8568-9baad12814c4\") "
Sep 08 13:35:24.042473 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.042451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f87b85ed-af72-4355-8568-9baad12814c4" (UID: "f87b85ed-af72-4355-8568-9baad12814c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:24.044340 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.044321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f87b85ed-af72-4355-8568-9baad12814c4" (UID: "f87b85ed-af72-4355-8568-9baad12814c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:24.044390 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.044360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v" (OuterVolumeSpecName: "kube-api-access-mns2v") pod "f87b85ed-af72-4355-8568-9baad12814c4" (UID: "f87b85ed-af72-4355-8568-9baad12814c4"). InnerVolumeSpecName "kube-api-access-mns2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:24.081547 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.081521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:35:24.143722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.143691    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mns2v\" (UniqueName: \"kubernetes.io/projected/f87b85ed-af72-4355-8568-9baad12814c4-kube-api-access-mns2v\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:24.143722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.143722    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f87b85ed-af72-4355-8568-9baad12814c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:24.143863 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.143732    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f87b85ed-af72-4355-8568-9baad12814c4-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:24.353622 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.353596    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:35:24.355612 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:35:24.355581    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ce2d85_13dd_4df7_aced_bb19cb150d51.slice/crio-bb24b9a23d4b7352dfcf04129b61feef11726d8d87e255f58a6eb4f9685f3d2c WatchSource:0}: Error finding container bb24b9a23d4b7352dfcf04129b61feef11726d8d87e255f58a6eb4f9685f3d2c: Status 404 returned error can't find the container with id bb24b9a23d4b7352dfcf04129b61feef11726d8d87e255f58a6eb4f9685f3d2c
Sep 08 13:35:24.601160 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.601122    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc357685-7683-4c3f-ad85-a55fc7789625" containerID="3cccd85b1da6acefd91bc704b13e77d90b74d72758a5db98c2700c928d4bcd66" exitCode=2
Sep 08 13:35:24.601344 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.601204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerDied","Data":"3cccd85b1da6acefd91bc704b13e77d90b74d72758a5db98c2700c928d4bcd66"}
Sep 08 13:35:24.602900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.602840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerStarted","Data":"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"}
Sep 08 13:35:24.602900 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.602890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerStarted","Data":"bb24b9a23d4b7352dfcf04129b61feef11726d8d87e255f58a6eb4f9685f3d2c"}
Sep 08 13:35:24.604621 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604598    2580 generic.go:356] "Generic (PLEG): container finished" podID="f87b85ed-af72-4355-8568-9baad12814c4" containerID="b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6" exitCode=2
Sep 08 13:35:24.604730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604624    2580 generic.go:356] "Generic (PLEG): container finished" podID="f87b85ed-af72-4355-8568-9baad12814c4" containerID="1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca" exitCode=2
Sep 08 13:35:24.604730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"
Sep 08 13:35:24.604730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerDied","Data":"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"}
Sep 08 13:35:24.604730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerDied","Data":"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"}
Sep 08 13:35:24.604875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n" event={"ID":"f87b85ed-af72-4355-8568-9baad12814c4","Type":"ContainerDied","Data":"ee01bb03f2562b63eacb76ffd2d64a9231a97bd81cad85c232d9c8a6c5946168"}
Sep 08 13:35:24.604875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.604777    2580 scope.go:117] "RemoveContainer" containerID="b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"
Sep 08 13:35:24.615017 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.614996    2580 scope.go:117] "RemoveContainer" containerID="1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"
Sep 08 13:35:24.626757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.626738    2580 scope.go:117] "RemoveContainer" containerID="b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"
Sep 08 13:35:24.627352 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:35:24.627328    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6\": container with ID starting with b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6 not found: ID does not exist" containerID="b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"
Sep 08 13:35:24.627458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627364    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"} err="failed to get container status \"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6\": rpc error: code = NotFound desc = could not find container \"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6\": container with ID starting with b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6 not found: ID does not exist"
Sep 08 13:35:24.627458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627388    2580 scope.go:117] "RemoveContainer" containerID="1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"
Sep 08 13:35:24.627670 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:35:24.627651    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca\": container with ID starting with 1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca not found: ID does not exist" containerID="1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"
Sep 08 13:35:24.627743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627674    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"} err="failed to get container status \"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca\": rpc error: code = NotFound desc = could not find container \"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca\": container with ID starting with 1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca not found: ID does not exist"
Sep 08 13:35:24.627743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627701    2580 scope.go:117] "RemoveContainer" containerID="b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"
Sep 08 13:35:24.627955 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627931    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6"} err="failed to get container status \"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6\": rpc error: code = NotFound desc = could not find container \"b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6\": container with ID starting with b2b2d86a7298600065959f4930faaf419eee05b6784ddfdc8d79c3e6343998c6 not found: ID does not exist"
Sep 08 13:35:24.628017 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.627956    2580 scope.go:117] "RemoveContainer" containerID="1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"
Sep 08 13:35:24.628197 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.628159    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca"} err="failed to get container status \"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca\": rpc error: code = NotFound desc = could not find container \"1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca\": container with ID starting with 1f88741b9369e2f74f6b97d1a815ba675523022c1bebda4abec96fae57c053ca not found: ID does not exist"
Sep 08 13:35:24.637459 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.637435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:35:24.643458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:24.643434    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2mc9n"]
Sep 08 13:35:26.536676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:26.536632    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87b85ed-af72-4355-8568-9baad12814c4" path="/var/lib/kubelet/pods/f87b85ed-af72-4355-8568-9baad12814c4/volumes"
Sep 08 13:35:27.169876 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:27.169830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:28.627209 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:28.627158    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc357685-7683-4c3f-ad85-a55fc7789625" containerID="12d7a50da7aea34f4ce704281f0fc1eca1d79cd8751c1c9ef6c916136cac7334" exitCode=0
Sep 08 13:35:28.627566 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:28.627230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerDied","Data":"12d7a50da7aea34f4ce704281f0fc1eca1d79cd8751c1c9ef6c916136cac7334"}
Sep 08 13:35:29.633496 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:29.633408    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerID="bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430" exitCode=0
Sep 08 13:35:29.633496 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:29.633480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerDied","Data":"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"}
Sep 08 13:35:29.634720 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:29.634705    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:35:32.169573 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:32.169527    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:32.175097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:32.175062    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:35:32.175519 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:32.175491    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:37.169155 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:37.169080    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:37.169673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:37.169247    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:42.170230 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:42.170184    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:42.175064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:42.175023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:35:42.175503 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:42.175474    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:47.169225 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:47.169158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:52.169501 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:52.169456    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 08 13:35:52.175979 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:52.175910    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 08 13:35:52.176240 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:52.176214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:52.176376 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:52.176351    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:52.176500 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:52.176482    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:53.773465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:53.773423    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc357685-7683-4c3f-ad85-a55fc7789625" containerID="a93ac01050a85b11a8d2aeab5c2a44895c7a975bc17d4ed2c49f09ee930953b5" exitCode=0
Sep 08 13:35:53.773886 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:53.773471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerDied","Data":"a93ac01050a85b11a8d2aeab5c2a44895c7a975bc17d4ed2c49f09ee930953b5"}
Sep 08 13:35:54.248506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.248470    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:54.314583 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.314553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxphg\" (UniqueName: \"kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg\") pod \"bc357685-7683-4c3f-ad85-a55fc7789625\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") "
Sep 08 13:35:54.314772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.314605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert\") pod \"bc357685-7683-4c3f-ad85-a55fc7789625\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") "
Sep 08 13:35:54.314772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.314635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config\") pod \"bc357685-7683-4c3f-ad85-a55fc7789625\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") "
Sep 08 13:35:54.314772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.314671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") pod \"bc357685-7683-4c3f-ad85-a55fc7789625\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") "
Sep 08 13:35:54.314772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.314703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location\") pod \"bc357685-7683-4c3f-ad85-a55fc7789625\" (UID: \"bc357685-7683-4c3f-ad85-a55fc7789625\") "
Sep 08 13:35:54.315294 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.315185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc357685-7683-4c3f-ad85-a55fc7789625" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:54.315294 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.315250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc357685-7683-4c3f-ad85-a55fc7789625" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:54.315465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.315407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc357685-7683-4c3f-ad85-a55fc7789625" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:54.317181 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.317139    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc357685-7683-4c3f-ad85-a55fc7789625" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:54.317348 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.317313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg" (OuterVolumeSpecName: "kube-api-access-pxphg") pod "bc357685-7683-4c3f-ad85-a55fc7789625" (UID: "bc357685-7683-4c3f-ad85-a55fc7789625"). InnerVolumeSpecName "kube-api-access-pxphg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:54.416281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.416199    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxphg\" (UniqueName: \"kubernetes.io/projected/bc357685-7683-4c3f-ad85-a55fc7789625-kube-api-access-pxphg\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.416281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.416235    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.416281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.416250    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc357685-7683-4c3f-ad85-a55fc7789625-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.416281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.416265    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc357685-7683-4c3f-ad85-a55fc7789625-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.416281 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.416282    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc357685-7683-4c3f-ad85-a55fc7789625-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.779404 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.779368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6" event={"ID":"bc357685-7683-4c3f-ad85-a55fc7789625","Type":"ContainerDied","Data":"61a016d276254903c48362a5962f47777619053ca8a3cfe9c288c7f30e8cda91"}
Sep 08 13:35:54.779833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.779455    2580 scope.go:117] "RemoveContainer" containerID="a93ac01050a85b11a8d2aeab5c2a44895c7a975bc17d4ed2c49f09ee930953b5"
Sep 08 13:35:54.779833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.779380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"
Sep 08 13:35:54.787580 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.787561    2580 scope.go:117] "RemoveContainer" containerID="3cccd85b1da6acefd91bc704b13e77d90b74d72758a5db98c2700c928d4bcd66"
Sep 08 13:35:54.794872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.794854    2580 scope.go:117] "RemoveContainer" containerID="12d7a50da7aea34f4ce704281f0fc1eca1d79cd8751c1c9ef6c916136cac7334"
Sep 08 13:35:54.801727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.801705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:35:54.802494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.802482    2580 scope.go:117] "RemoveContainer" containerID="e6572fa48577ea49362ee709640ed12bc72454b32561cccd83d7acea06a54f93"
Sep 08 13:35:54.808118 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:54.808097    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5979d78599-hlwv6"]
Sep 08 13:35:56.537724 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:35:56.537688    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" path="/var/lib/kubelet/pods/bc357685-7683-4c3f-ad85-a55fc7789625/volumes"
Sep 08 13:37:19.007303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:19.007276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:37:19.007846 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:19.007801    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:37:20.235930 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.235890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerStarted","Data":"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"}
Sep 08 13:37:20.236365 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.235991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerStarted","Data":"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"}
Sep 08 13:37:20.236365 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.236030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:20.236365 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.236060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:20.237249 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.237225    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 08 13:37:20.263947 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:20.263894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podStartSLOduration=2.404506816 podStartE2EDuration="1m57.263879579s" totalImagesPullingTime="1m49.922639195s" totalInitContainerRuntime="4.936733568s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:37:20.262905054 +0000 UTC m=+1236.213559005" watchObservedRunningTime="2026-09-08 13:37:20.263879579 +0000 UTC m=+1236.214533532"
Sep 08 13:37:21.242015 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:21.241967    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 08 13:37:26.247070 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:26.247035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:26.247686 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:26.247507    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 08 13:37:36.248326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:36.248292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:49.875211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:49.875178    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:37:49.875668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:49.875448    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container" containerID="cri-o://5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992" gracePeriod=30
Sep 08 13:37:49.875668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:49.875478    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kube-rbac-proxy" containerID="cri-o://822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b" gracePeriod=30
Sep 08 13:37:50.380553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:50.380521    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerID="822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b" exitCode=2
Sep 08 13:37:50.380735 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:50.380594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerDied","Data":"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"}
Sep 08 13:37:51.254222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.254195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:51.358234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls\") pod \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") "
Sep 08 13:37:51.358234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location\") pod \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") "
Sep 08 13:37:51.358405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhw5r\" (UniqueName: \"kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r\") pod \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") "
Sep 08 13:37:51.358447 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert\") pod \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") "
Sep 08 13:37:51.358481 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config\") pod \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\" (UID: \"d3ce2d85-13dd-4df7-aced-bb19cb150d51\") "
Sep 08 13:37:51.358825 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3ce2d85-13dd-4df7-aced-bb19cb150d51" (UID: "d3ce2d85-13dd-4df7-aced-bb19cb150d51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:51.358973 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.358895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3ce2d85-13dd-4df7-aced-bb19cb150d51" (UID: "d3ce2d85-13dd-4df7-aced-bb19cb150d51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:51.360280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.360260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3ce2d85-13dd-4df7-aced-bb19cb150d51" (UID: "d3ce2d85-13dd-4df7-aced-bb19cb150d51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:51.360530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.360505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r" (OuterVolumeSpecName: "kube-api-access-vhw5r") pod "d3ce2d85-13dd-4df7-aced-bb19cb150d51" (UID: "d3ce2d85-13dd-4df7-aced-bb19cb150d51"). InnerVolumeSpecName "kube-api-access-vhw5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:51.362814 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.362790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3ce2d85-13dd-4df7-aced-bb19cb150d51" (UID: "d3ce2d85-13dd-4df7-aced-bb19cb150d51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:51.387004 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.386980    2580 generic.go:356] "Generic (PLEG): container finished" podID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerID="5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992" exitCode=0
Sep 08 13:37:51.387126 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.387044    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"
Sep 08 13:37:51.387126 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.387055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerDied","Data":"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"}
Sep 08 13:37:51.387126 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.387109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5" event={"ID":"d3ce2d85-13dd-4df7-aced-bb19cb150d51","Type":"ContainerDied","Data":"bb24b9a23d4b7352dfcf04129b61feef11726d8d87e255f58a6eb4f9685f3d2c"}
Sep 08 13:37:51.387312 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.387138    2580 scope.go:117] "RemoveContainer" containerID="822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"
Sep 08 13:37:51.395867 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.395851    2580 scope.go:117] "RemoveContainer" containerID="5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"
Sep 08 13:37:51.403359 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.403345    2580 scope.go:117] "RemoveContainer" containerID="bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"
Sep 08 13:37:51.410612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.410596    2580 scope.go:117] "RemoveContainer" containerID="822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"
Sep 08 13:37:51.410923 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:37:51.410897    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b\": container with ID starting with 822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b not found: ID does not exist" containerID="822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"
Sep 08 13:37:51.410984 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.410936    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b"} err="failed to get container status \"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b\": rpc error: code = NotFound desc = could not find container \"822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b\": container with ID starting with 822712e479906ef4ef5fe8606a3e5acbe7655dbcc19f3bd8b0ca91f765cffe4b not found: ID does not exist"
Sep 08 13:37:51.410984 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.410958    2580 scope.go:117] "RemoveContainer" containerID="5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"
Sep 08 13:37:51.411260 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:37:51.411242    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992\": container with ID starting with 5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992 not found: ID does not exist" containerID="5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"
Sep 08 13:37:51.411326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.411264    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992"} err="failed to get container status \"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992\": rpc error: code = NotFound desc = could not find container \"5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992\": container with ID starting with 5a8d02a7077524622f3fd2886bf8cf65e9bcd7d63737905031239674b3dc7992 not found: ID does not exist"
Sep 08 13:37:51.411326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.411281    2580 scope.go:117] "RemoveContainer" containerID="bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"
Sep 08 13:37:51.411540 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:37:51.411524    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430\": container with ID starting with bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430 not found: ID does not exist" containerID="bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"
Sep 08 13:37:51.411590 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.411544    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430"} err="failed to get container status \"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430\": rpc error: code = NotFound desc = could not find container \"bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430\": container with ID starting with bb96a7b07341b4c01006cca9b47d8f6fb60a2c579d7d946d259464126a6a8430 not found: ID does not exist"
Sep 08 13:37:51.416299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.416275    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:37:51.418001 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.417981    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-vgvg5"]
Sep 08 13:37:51.459363 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.459341    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ce2d85-13dd-4df7-aced-bb19cb150d51-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:37:51.459363 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.459360    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:37:51.459487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.459371    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhw5r\" (UniqueName: \"kubernetes.io/projected/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-api-access-vhw5r\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:37:51.459487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.459381    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:37:51.459487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:51.459390    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ce2d85-13dd-4df7-aced-bb19cb150d51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:37:52.535567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:52.535534    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" path="/var/lib/kubelet/pods/d3ce2d85-13dd-4df7-aced-bb19cb150d51/volumes"
Sep 08 13:37:55.139335 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139297    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:37:55.139723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139705    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="storage-initializer"
Sep 08 13:37:55.139723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139718    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="storage-initializer"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139734    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kube-rbac-proxy"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139749    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kube-rbac-proxy"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139757    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kserve-container"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139762    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kserve-container"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139777    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139795    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139801    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="storage-initializer"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="storage-initializer"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kube-rbac-proxy"
Sep 08 13:37:55.139817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139817    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kube-rbac-proxy"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139827    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139832    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139885    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="agent"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139891    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kserve-container"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kube-rbac-proxy"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139904    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87b85ed-af72-4355-8568-9baad12814c4" containerName="kserve-container"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139912    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ce2d85-13dd-4df7-aced-bb19cb150d51" containerName="kube-rbac-proxy"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139917    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kube-rbac-proxy"
Sep 08 13:37:55.140301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.139921    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc357685-7683-4c3f-ad85-a55fc7789625" containerName="kserve-container"
Sep 08 13:37:55.164682 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.164653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:37:55.164847 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.164764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.167757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.167731    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:37:55.167757 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.167753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:37:55.167949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.167795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:37:55.167949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.167925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 08 13:37:55.168815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.168785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:37:55.168815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.168809    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 08 13:37:55.168983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.168836    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:37:55.289217 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.289183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmdd\" (UniqueName: \"kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.289384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.289230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.289384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.289255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.289384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.289303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.289384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.289342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.390779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.390705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.390779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.390745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.390779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.390767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.390991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.390881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.390991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.390971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmdd\" (UniqueName: \"kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.391227 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.391207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.391456 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.391436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.391532 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.391501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.393255 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.393235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.400462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.400443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmdd\" (UniqueName: \"kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xgjdw\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.474551 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.474525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:37:55.773431 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:55.773406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:37:55.775360 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:37:55.775327    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188e0d1c_be88_4ef0_beb2_64e2589437f0.slice/crio-37d7583c24a33bd1e3cfd2894a66fb4e13d3460c3c2b657cea52e127c4bd5a93 WatchSource:0}: Error finding container 37d7583c24a33bd1e3cfd2894a66fb4e13d3460c3c2b657cea52e127c4bd5a93: Status 404 returned error can't find the container with id 37d7583c24a33bd1e3cfd2894a66fb4e13d3460c3c2b657cea52e127c4bd5a93
Sep 08 13:37:56.414097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:56.414057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerStarted","Data":"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"}
Sep 08 13:37:56.414550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:37:56.414113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerStarted","Data":"37d7583c24a33bd1e3cfd2894a66fb4e13d3460c3c2b657cea52e127c4bd5a93"}
Sep 08 13:38:01.437738 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:01.437704    2580 generic.go:356] "Generic (PLEG): container finished" podID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerID="d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1" exitCode=0
Sep 08 13:38:01.438229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:01.437779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerDied","Data":"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"}
Sep 08 13:38:02.446387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:02.446352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerStarted","Data":"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"}
Sep 08 13:38:02.446812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:02.446414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerStarted","Data":"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"}
Sep 08 13:38:02.446812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:02.446606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:38:02.446812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:02.446652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:38:02.472502 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:02.472452    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" podStartSLOduration=2.80197249 podStartE2EDuration="7.472435801s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.670463311s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:02.470413007 +0000 UTC m=+1278.421066958" watchObservedRunningTime="2026-09-08 13:38:02.472435801 +0000 UTC m=+1278.423089753"
Sep 08 13:38:08.458786 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:38:08.458746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:39:40.539178 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:40.539141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:39:51.208837 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:51.208803    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:39:51.209233 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:51.209090    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kserve-container" containerID="cri-o://ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13" gracePeriod=30
Sep 08 13:39:51.209233 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:51.209120    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kube-rbac-proxy" containerID="cri-o://edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694" gracePeriod=30
Sep 08 13:39:51.991396 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:51.991358    2580 generic.go:356] "Generic (PLEG): container finished" podID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerID="edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694" exitCode=2
Sep 08 13:39:51.991581 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:51.991436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerDied","Data":"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"}
Sep 08 13:39:52.558552 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.558529    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:39:52.650738 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.650702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config\") pod \"188e0d1c-be88-4ef0-beb2-64e2589437f0\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") "
Sep 08 13:39:52.650738 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.650741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location\") pod \"188e0d1c-be88-4ef0-beb2-64e2589437f0\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") "
Sep 08 13:39:52.650931 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.650771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hmdd\" (UniqueName: \"kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd\") pod \"188e0d1c-be88-4ef0-beb2-64e2589437f0\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") "
Sep 08 13:39:52.650931 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.650819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert\") pod \"188e0d1c-be88-4ef0-beb2-64e2589437f0\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") "
Sep 08 13:39:52.651076 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.651049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls\") pod \"188e0d1c-be88-4ef0-beb2-64e2589437f0\" (UID: \"188e0d1c-be88-4ef0-beb2-64e2589437f0\") "
Sep 08 13:39:52.651153 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.651129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "188e0d1c-be88-4ef0-beb2-64e2589437f0" (UID: "188e0d1c-be88-4ef0-beb2-64e2589437f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:52.651245 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.651184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "188e0d1c-be88-4ef0-beb2-64e2589437f0" (UID: "188e0d1c-be88-4ef0-beb2-64e2589437f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:52.651405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.651374    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.651405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.651398    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/188e0d1c-be88-4ef0-beb2-64e2589437f0-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.653073 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.653018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd" (OuterVolumeSpecName: "kube-api-access-7hmdd") pod "188e0d1c-be88-4ef0-beb2-64e2589437f0" (UID: "188e0d1c-be88-4ef0-beb2-64e2589437f0"). InnerVolumeSpecName "kube-api-access-7hmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:52.653183 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.653126    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "188e0d1c-be88-4ef0-beb2-64e2589437f0" (UID: "188e0d1c-be88-4ef0-beb2-64e2589437f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:39:52.657062 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.657033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "188e0d1c-be88-4ef0-beb2-64e2589437f0" (UID: "188e0d1c-be88-4ef0-beb2-64e2589437f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:39:52.751872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.751844    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/188e0d1c-be88-4ef0-beb2-64e2589437f0-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.751872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.751869    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmdd\" (UniqueName: \"kubernetes.io/projected/188e0d1c-be88-4ef0-beb2-64e2589437f0-kube-api-access-7hmdd\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.751872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.751879    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188e0d1c-be88-4ef0-beb2-64e2589437f0-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.998258 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.998222    2580 generic.go:356] "Generic (PLEG): container finished" podID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerID="ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13" exitCode=0
Sep 08 13:39:52.998440 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.998278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"
Sep 08 13:39:52.998440 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.998304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerDied","Data":"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"}
Sep 08 13:39:52.998440 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.998367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw" event={"ID":"188e0d1c-be88-4ef0-beb2-64e2589437f0","Type":"ContainerDied","Data":"37d7583c24a33bd1e3cfd2894a66fb4e13d3460c3c2b657cea52e127c4bd5a93"}
Sep 08 13:39:52.998440 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:52.998393    2580 scope.go:117] "RemoveContainer" containerID="edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"
Sep 08 13:39:53.007392 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.007375    2580 scope.go:117] "RemoveContainer" containerID="ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"
Sep 08 13:39:53.014953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.014928    2580 scope.go:117] "RemoveContainer" containerID="d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"
Sep 08 13:39:53.022191 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022159    2580 scope.go:117] "RemoveContainer" containerID="edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"
Sep 08 13:39:53.022381 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:39:53.022468 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:39:53.022451    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694\": container with ID starting with edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694 not found: ID does not exist" containerID="edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"
Sep 08 13:39:53.022511 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022477    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694"} err="failed to get container status \"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694\": rpc error: code = NotFound desc = could not find container \"edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694\": container with ID starting with edfef26f4d809cac3ce2052f57130ab8ef3e777bd503a17771b77b937997c694 not found: ID does not exist"
Sep 08 13:39:53.022511 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022497    2580 scope.go:117] "RemoveContainer" containerID="ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"
Sep 08 13:39:53.022756 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:39:53.022733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13\": container with ID starting with ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13 not found: ID does not exist" containerID="ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"
Sep 08 13:39:53.022801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022753    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13"} err="failed to get container status \"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13\": rpc error: code = NotFound desc = could not find container \"ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13\": container with ID starting with ded53891efb3816ab0b141f4068eb546611ee64565a1de715cd390c509c3ba13 not found: ID does not exist"
Sep 08 13:39:53.022801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.022782    2580 scope.go:117] "RemoveContainer" containerID="d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"
Sep 08 13:39:53.022997 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:39:53.022982    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1\": container with ID starting with d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1 not found: ID does not exist" containerID="d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"
Sep 08 13:39:53.023034 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.023000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1"} err="failed to get container status \"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1\": rpc error: code = NotFound desc = could not find container \"d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1\": container with ID starting with d3b217aca38346945bfb848b67ba0169c469639f20d59c2470e70670c2b1aac1 not found: ID does not exist"
Sep 08 13:39:53.026673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:53.026650    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xgjdw"]
Sep 08 13:39:54.535844 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:54.535810    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" path="/var/lib/kubelet/pods/188e0d1c-be88-4ef0-beb2-64e2589437f0/volumes"
Sep 08 13:39:56.451998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.451962    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:39:56.452386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452333    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kserve-container"
Sep 08 13:39:56.452386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kserve-container"
Sep 08 13:39:56.452386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452356    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kube-rbac-proxy"
Sep 08 13:39:56.452386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452362    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kube-rbac-proxy"
Sep 08 13:39:56.452386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452384    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="storage-initializer"
Sep 08 13:39:56.452545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452391    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="storage-initializer"
Sep 08 13:39:56.452545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452451    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kube-rbac-proxy"
Sep 08 13:39:56.452545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.452463    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="188e0d1c-be88-4ef0-beb2-64e2589437f0" containerName="kserve-container"
Sep 08 13:39:56.455625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.455600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.458382 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.458361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 08 13:39:56.458506 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.458368    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:39:56.458856 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.458841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:39:56.458916 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.458882    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 08 13:39:56.459558 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.459539    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:39:56.459612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.459596    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:39:56.459821 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.459807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:39:56.466502 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.466482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:39:56.582870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.582836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.583032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.582880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.583032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.582961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.583032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.583005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.583032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.583030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgpz\" (UniqueName: \"kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgpz\" (UniqueName: \"kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684248 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684378 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684651 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684926 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.684963 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.684905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.686628 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.686601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.694581 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.694555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgpz\" (UniqueName: \"kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:56.766627 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:56.766581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:39:57.043976 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:57.043953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:39:57.045736 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:39:57.045709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f46a15_cba0_4fb6_a682_e0fe8b3ed64b.slice/crio-405fdb075d33d5eb37c3b9cf7069fe489cd21b0208dbdd4421ab09e42d4e585a WatchSource:0}: Error finding container 405fdb075d33d5eb37c3b9cf7069fe489cd21b0208dbdd4421ab09e42d4e585a: Status 404 returned error can't find the container with id 405fdb075d33d5eb37c3b9cf7069fe489cd21b0208dbdd4421ab09e42d4e585a
Sep 08 13:39:58.031642 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:58.031603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerStarted","Data":"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"}
Sep 08 13:39:58.032011 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:39:58.031665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerStarted","Data":"405fdb075d33d5eb37c3b9cf7069fe489cd21b0208dbdd4421ab09e42d4e585a"}
Sep 08 13:40:26.166222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:26.166116    2580 generic.go:356] "Generic (PLEG): container finished" podID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerID="1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316" exitCode=0
Sep 08 13:40:26.166222 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:26.166195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerDied","Data":"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"}
Sep 08 13:40:27.173790 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:27.173746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerStarted","Data":"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"}
Sep 08 13:40:27.174186 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:27.173836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerStarted","Data":"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"}
Sep 08 13:40:27.174186 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:27.174045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:40:27.227220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:27.227141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" podStartSLOduration=2.46252832 podStartE2EDuration="31.227117627s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.764589307s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:27.221947311 +0000 UTC m=+1423.172601262" watchObservedRunningTime="2026-09-08 13:40:27.227117627 +0000 UTC m=+1423.177771582"
Sep 08 13:40:28.178226 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:28.178190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:40:34.187883 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:40:34.187846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:41:14.615554 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.615518    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:41:14.620589 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.620572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.623259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.623239    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 08 13:41:14.623541 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.623522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 08 13:41:14.633094 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.633074    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:41:14.645376 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.645356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.645472 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.645387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.645472 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.645426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zlr\" (UniqueName: \"kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.645567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.645504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.645567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.645552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746621 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.746588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zlr\" (UniqueName: \"kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.746629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.746673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.746726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.746758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.746960 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:41:14.746854    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 08 13:41:14.746960 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:41:14.746928    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls podName:2d11d2ca-8154-4cfc-ac56-e81cda90f79d nodeName:}" failed. No retries permitted until 2026-09-08 13:41:15.246909244 +0000 UTC m=+1471.197563188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls") pod "isvc-sklearn-predictor-8fdc756d-x74vf" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 08 13:41:14.747141 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.747120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.747432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.747413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.747479 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.747423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:14.755791 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:14.755768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zlr\" (UniqueName: \"kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:15.251474 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:15.251429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:15.251645 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:41:15.251586    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 08 13:41:15.251710 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:41:15.251656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls podName:2d11d2ca-8154-4cfc-ac56-e81cda90f79d nodeName:}" failed. No retries permitted until 2026-09-08 13:41:16.251640618 +0000 UTC m=+1472.202294548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls") pod "isvc-sklearn-predictor-8fdc756d-x74vf" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 08 13:41:16.260301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:16.260267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:16.262737 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:16.262716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"isvc-sklearn-predictor-8fdc756d-x74vf\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:16.430747 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:16.430719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:16.708373 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:16.708351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:41:16.710476 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:41:16.710452    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d11d2ca_8154_4cfc_ac56_e81cda90f79d.slice/crio-810a88e8ebed125f6f159797c0611ec4eb4e21823909b6398b75b9a03d001322 WatchSource:0}: Error finding container 810a88e8ebed125f6f159797c0611ec4eb4e21823909b6398b75b9a03d001322: Status 404 returned error can't find the container with id 810a88e8ebed125f6f159797c0611ec4eb4e21823909b6398b75b9a03d001322
Sep 08 13:41:17.407612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:17.407576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerStarted","Data":"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"}
Sep 08 13:41:17.407990 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:17.407635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerStarted","Data":"810a88e8ebed125f6f159797c0611ec4eb4e21823909b6398b75b9a03d001322"}
Sep 08 13:41:21.427559 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:21.427525    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerID="92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac" exitCode=0
Sep 08 13:41:21.427984 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:21.427601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerDied","Data":"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"}
Sep 08 13:41:22.434960 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:22.434926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerStarted","Data":"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"}
Sep 08 13:41:22.435372 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:22.434989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerStarted","Data":"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"}
Sep 08 13:41:22.435372 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:22.435192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:23.439957 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:23.439922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:23.441389 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:23.441362    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:41:24.444799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:24.444756    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:41:29.449141 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:29.449110    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:41:29.449749 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:29.449723    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:41:29.473346 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:29.473295    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podStartSLOduration=11.688306363 podStartE2EDuration="15.473279254s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784972891s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:22.458256147 +0000 UTC m=+1478.408910100" watchObservedRunningTime="2026-09-08 13:41:29.473279254 +0000 UTC m=+1485.423933206"
Sep 08 13:41:39.450527 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:39.450482    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:41:49.450722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:49.450681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:41:59.450570 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:41:59.450532    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:42:02.536833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:02.536803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:42:09.450092 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:09.450004    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:42:13.074087 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:13.074052    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:42:13.074559 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:13.074376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kube-rbac-proxy" containerID="cri-o://dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1" gracePeriod=30
Sep 08 13:42:13.074559 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:13.074365    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kserve-container" containerID="cri-o://e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e" gracePeriod=30
Sep 08 13:42:13.690609 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:13.690575    2580 generic.go:356] "Generic (PLEG): container finished" podID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerID="dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1" exitCode=2
Sep 08 13:42:13.690779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:13.690630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerDied","Data":"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"}
Sep 08 13:42:14.183295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:14.183250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 08 13:42:18.119948 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.119925    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:42:18.174288 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccgpz\" (UniqueName: \"kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz\") pod \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") "
Sep 08 13:42:18.174452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location\") pod \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") "
Sep 08 13:42:18.174452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config\") pod \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") "
Sep 08 13:42:18.174452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert\") pod \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") "
Sep 08 13:42:18.174452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls\") pod \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\" (UID: \"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b\") "
Sep 08 13:42:18.174786 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" (UID: "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:42:18.174886 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.174755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" (UID: "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:42:18.176377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.176354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz" (OuterVolumeSpecName: "kube-api-access-ccgpz") pod "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" (UID: "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b"). InnerVolumeSpecName "kube-api-access-ccgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:42:18.176953 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.176927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" (UID: "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:42:18.179880 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.179857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" (UID: "27f46a15-cba0-4fb6-a682-e0fe8b3ed64b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:42:18.275278 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.275233    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgpz\" (UniqueName: \"kubernetes.io/projected/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-api-access-ccgpz\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:18.275278 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.275272    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:18.275278 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.275282    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:18.275278 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.275291    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:18.275551 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.275300    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:18.306426 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306393    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:42:18.306762 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306748    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kserve-container"
Sep 08 13:42:18.306815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306764    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kserve-container"
Sep 08 13:42:18.306815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kube-rbac-proxy"
Sep 08 13:42:18.306815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306783    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kube-rbac-proxy"
Sep 08 13:42:18.306815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="storage-initializer"
Sep 08 13:42:18.306815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306795    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="storage-initializer"
Sep 08 13:42:18.306969 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306853    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kserve-container"
Sep 08 13:42:18.306969 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.306863    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerName="kube-rbac-proxy"
Sep 08 13:42:18.311526 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.311508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.314746 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.314726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 08 13:42:18.314860 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.314843    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 08 13:42:18.329477 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.329453    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:42:18.376264 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.376220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.376419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.376283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.376419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.376329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.376419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.376364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlkd\" (UniqueName: \"kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.376419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.376404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.477576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.477537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.477742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.477642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.477742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.477715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.477865 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.477748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlkd\" (UniqueName: \"kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.477865 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.477789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.478123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.478103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.478291 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.478274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.478341 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.478295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.480176 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.480148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.490531 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.490505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlkd\" (UniqueName: \"kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.621754 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.621724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:18.722761 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.722724    2580 generic.go:356] "Generic (PLEG): container finished" podID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" containerID="e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e" exitCode=0
Sep 08 13:42:18.722974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.722789    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"
Sep 08 13:42:18.722974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.722799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerDied","Data":"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"}
Sep 08 13:42:18.722974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.722887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8" event={"ID":"27f46a15-cba0-4fb6-a682-e0fe8b3ed64b","Type":"ContainerDied","Data":"405fdb075d33d5eb37c3b9cf7069fe489cd21b0208dbdd4421ab09e42d4e585a"}
Sep 08 13:42:18.722974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.722917    2580 scope.go:117] "RemoveContainer" containerID="dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"
Sep 08 13:42:18.731297 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.731279    2580 scope.go:117] "RemoveContainer" containerID="e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"
Sep 08 13:42:18.739624 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.739600    2580 scope.go:117] "RemoveContainer" containerID="1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"
Sep 08 13:42:18.747310 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747293    2580 scope.go:117] "RemoveContainer" containerID="dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"
Sep 08 13:42:18.747631 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:18.747611    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1\": container with ID starting with dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1 not found: ID does not exist" containerID="dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"
Sep 08 13:42:18.747697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747653    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1"} err="failed to get container status \"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1\": rpc error: code = NotFound desc = could not find container \"dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1\": container with ID starting with dae85d01d82db6da7db361f8955e4eadeb703066712a16493e41b68641e71ba1 not found: ID does not exist"
Sep 08 13:42:18.747697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747672    2580 scope.go:117] "RemoveContainer" containerID="e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"
Sep 08 13:42:18.747789 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:42:18.747941 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:18.747924    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e\": container with ID starting with e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e not found: ID does not exist" containerID="e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"
Sep 08 13:42:18.747982 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747947    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e"} err="failed to get container status \"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e\": rpc error: code = NotFound desc = could not find container \"e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e\": container with ID starting with e26f9d1aa1b7bf10cc734958f0dd5aa6f92ad2b62ab2cc6ac97227173cbb1b8e not found: ID does not exist"
Sep 08 13:42:18.747982 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.747967    2580 scope.go:117] "RemoveContainer" containerID="1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"
Sep 08 13:42:18.748220 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:18.748154    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316\": container with ID starting with 1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316 not found: ID does not exist" containerID="1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"
Sep 08 13:42:18.748220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.748193    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316"} err="failed to get container status \"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316\": rpc error: code = NotFound desc = could not find container \"1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316\": container with ID starting with 1a31597a201459884cc9af95d0f54591dbf96e68843b7e476d228f9bd2a1d316 not found: ID does not exist"
Sep 08 13:42:18.755350 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.755331    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vvx8"]
Sep 08 13:42:18.902803 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:18.902624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:42:18.905604 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:42:18.905581    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9a4013_1007_4a98_b57b_24d79c682dce.slice/crio-8b2de8dd59afd5c21c450798be89bf13788e2371f2ace7bf219d146b7d59487a WatchSource:0}: Error finding container 8b2de8dd59afd5c21c450798be89bf13788e2371f2ace7bf219d146b7d59487a: Status 404 returned error can't find the container with id 8b2de8dd59afd5c21c450798be89bf13788e2371f2ace7bf219d146b7d59487a
Sep 08 13:42:19.044961 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:19.044934    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:42:19.045915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:19.045891    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:42:19.449980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:19.449936    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:42:19.729419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:19.729376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerStarted","Data":"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"}
Sep 08 13:42:19.729655 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:19.729451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerStarted","Data":"8b2de8dd59afd5c21c450798be89bf13788e2371f2ace7bf219d146b7d59487a"}
Sep 08 13:42:20.535960 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:20.535918    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f46a15-cba0-4fb6-a682-e0fe8b3ed64b" path="/var/lib/kubelet/pods/27f46a15-cba0-4fb6-a682-e0fe8b3ed64b/volumes"
Sep 08 13:42:29.450664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:29.450617    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 08 13:42:34.805299 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:34.805262    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerID="e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763" exitCode=0
Sep 08 13:42:34.805711 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:34.805339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerDied","Data":"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"}
Sep 08 13:42:35.813327 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:35.813289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerStarted","Data":"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"}
Sep 08 13:42:35.813799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:35.813358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerStarted","Data":"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"}
Sep 08 13:42:35.813799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:35.813585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:35.813799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:35.813630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:39.450393 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:39.450357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:42:39.476243 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:39.476181    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" podStartSLOduration=6.615028724 podStartE2EDuration="21.476147712s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.861118988s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:42:35.843160886 +0000 UTC m=+1551.793814839" watchObservedRunningTime="2026-09-08 13:42:39.476147712 +0000 UTC m=+1555.426801664"
Sep 08 13:42:41.825727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:41.825697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:42:44.714967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:44.714934    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:42:44.715385 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:44.715150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container" containerID="cri-o://5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65" gracePeriod=30
Sep 08 13:42:44.715385 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:44.715238    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kube-rbac-proxy" containerID="cri-o://79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c" gracePeriod=30
Sep 08 13:42:44.860719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:44.860689    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerID="79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c" exitCode=2
Sep 08 13:42:44.860864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:44.860726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerDied","Data":"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"}
Sep 08 13:42:49.358776 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.358750    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:42:49.447649 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location\") pod \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") "
Sep 08 13:42:49.447649 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert\") pod \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") "
Sep 08 13:42:49.447649 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") pod \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") "
Sep 08 13:42:49.447932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config\") pod \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") "
Sep 08 13:42:49.447932 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9zlr\" (UniqueName: \"kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr\") pod \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\" (UID: \"2d11d2ca-8154-4cfc-ac56-e81cda90f79d\") "
Sep 08 13:42:49.448024 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d11d2ca-8154-4cfc-ac56-e81cda90f79d" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:42:49.448024 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d11d2ca-8154-4cfc-ac56-e81cda90f79d" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:42:49.448024 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.447968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d11d2ca-8154-4cfc-ac56-e81cda90f79d" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:42:49.449845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.449817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d11d2ca-8154-4cfc-ac56-e81cda90f79d" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:42:49.449946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.449870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr" (OuterVolumeSpecName: "kube-api-access-x9zlr") pod "2d11d2ca-8154-4cfc-ac56-e81cda90f79d" (UID: "2d11d2ca-8154-4cfc-ac56-e81cda90f79d"). InnerVolumeSpecName "kube-api-access-x9zlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:42:49.549288 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.549258    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:49.549288 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.549282    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:49.549288 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.549291    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zlr\" (UniqueName: \"kubernetes.io/projected/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kube-api-access-x9zlr\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:49.549495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.549300    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:49.549495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.549309    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d11d2ca-8154-4cfc-ac56-e81cda90f79d-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:42:49.889259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.889223    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerID="5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65" exitCode=0
Sep 08 13:42:49.889494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.889290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"
Sep 08 13:42:49.889494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.889324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerDied","Data":"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"}
Sep 08 13:42:49.889494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.889423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf" event={"ID":"2d11d2ca-8154-4cfc-ac56-e81cda90f79d","Type":"ContainerDied","Data":"810a88e8ebed125f6f159797c0611ec4eb4e21823909b6398b75b9a03d001322"}
Sep 08 13:42:49.889494 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.889463    2580 scope.go:117] "RemoveContainer" containerID="79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"
Sep 08 13:42:49.898501 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.898478    2580 scope.go:117] "RemoveContainer" containerID="5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"
Sep 08 13:42:49.907529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.907508    2580 scope.go:117] "RemoveContainer" containerID="92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"
Sep 08 13:42:49.914827 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.914810    2580 scope.go:117] "RemoveContainer" containerID="79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"
Sep 08 13:42:49.915108 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:49.915089    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c\": container with ID starting with 79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c not found: ID does not exist" containerID="79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"
Sep 08 13:42:49.915153 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.915117    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c"} err="failed to get container status \"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c\": rpc error: code = NotFound desc = could not find container \"79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c\": container with ID starting with 79e4e386c7482e8caf8d194758a26c4b9ec3fc8dab7dd3b5a83ec80f70f3cc9c not found: ID does not exist"
Sep 08 13:42:49.915153 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.915135    2580 scope.go:117] "RemoveContainer" containerID="5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"
Sep 08 13:42:49.915397 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:49.915378    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65\": container with ID starting with 5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65 not found: ID does not exist" containerID="5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"
Sep 08 13:42:49.915452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.915401    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65"} err="failed to get container status \"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65\": rpc error: code = NotFound desc = could not find container \"5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65\": container with ID starting with 5b98f1989f634402d32f1d95e5c032fde437db6a3c099d3f2ee543b1f0e44e65 not found: ID does not exist"
Sep 08 13:42:49.915452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.915417    2580 scope.go:117] "RemoveContainer" containerID="92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"
Sep 08 13:42:49.915626 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:42:49.915609    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac\": container with ID starting with 92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac not found: ID does not exist" containerID="92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"
Sep 08 13:42:49.915668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.915630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac"} err="failed to get container status \"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac\": rpc error: code = NotFound desc = could not find container \"92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac\": container with ID starting with 92dda9e7f9e5082cc4da183db877d9a7939a9ece03ec0433eb2acae6ccee39ac not found: ID does not exist"
Sep 08 13:42:49.928295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.928273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:42:49.930445 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:49.930426    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-8fdc756d-x74vf"]
Sep 08 13:42:50.537069 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:42:50.537035    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" path="/var/lib/kubelet/pods/2d11d2ca-8154-4cfc-ac56-e81cda90f79d/volumes"
Sep 08 13:44:05.533928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:05.533897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:44:14.398568 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:14.398534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:44:14.398963 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:14.398781    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kserve-container" containerID="cri-o://1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a" gracePeriod=30
Sep 08 13:44:14.398963 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:14.398803    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kube-rbac-proxy" containerID="cri-o://2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d" gracePeriod=30
Sep 08 13:44:15.296812 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.296775    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerID="2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d" exitCode=2
Sep 08 13:44:15.297026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.296857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerDied","Data":"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"}
Sep 08 13:44:15.530573 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.530526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 08 13:44:15.745929 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.745907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:44:15.866981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.866897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls\") pod \"cc9a4013-1007-4a98-b57b-24d79c682dce\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") "
Sep 08 13:44:15.866981 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.866951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location\") pod \"cc9a4013-1007-4a98-b57b-24d79c682dce\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") "
Sep 08 13:44:15.867257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.866989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxlkd\" (UniqueName: \"kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd\") pod \"cc9a4013-1007-4a98-b57b-24d79c682dce\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") "
Sep 08 13:44:15.867257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.867031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert\") pod \"cc9a4013-1007-4a98-b57b-24d79c682dce\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") "
Sep 08 13:44:15.867257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.867062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config\") pod \"cc9a4013-1007-4a98-b57b-24d79c682dce\" (UID: \"cc9a4013-1007-4a98-b57b-24d79c682dce\") "
Sep 08 13:44:15.867459 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.867429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc9a4013-1007-4a98-b57b-24d79c682dce" (UID: "cc9a4013-1007-4a98-b57b-24d79c682dce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:44:15.867527 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.867502    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc9a4013-1007-4a98-b57b-24d79c682dce" (UID: "cc9a4013-1007-4a98-b57b-24d79c682dce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:44:15.869004 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.868983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc9a4013-1007-4a98-b57b-24d79c682dce" (UID: "cc9a4013-1007-4a98-b57b-24d79c682dce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:44:15.869107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.869086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd" (OuterVolumeSpecName: "kube-api-access-pxlkd") pod "cc9a4013-1007-4a98-b57b-24d79c682dce" (UID: "cc9a4013-1007-4a98-b57b-24d79c682dce"). InnerVolumeSpecName "kube-api-access-pxlkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:15.871700 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.871680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc9a4013-1007-4a98-b57b-24d79c682dce" (UID: "cc9a4013-1007-4a98-b57b-24d79c682dce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:15.968387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.968351    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc9a4013-1007-4a98-b57b-24d79c682dce-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.968387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.968382    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc9a4013-1007-4a98-b57b-24d79c682dce-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.968586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.968398    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlkd\" (UniqueName: \"kubernetes.io/projected/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-api-access-pxlkd\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.968586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.968411    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.968586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:15.968422    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc9a4013-1007-4a98-b57b-24d79c682dce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:44:16.302962 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.302923    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerID="1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a" exitCode=0
Sep 08 13:44:16.303133 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.302986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"
Sep 08 13:44:16.303133 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.302999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerDied","Data":"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"}
Sep 08 13:44:16.303133 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.303060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw" event={"ID":"cc9a4013-1007-4a98-b57b-24d79c682dce","Type":"ContainerDied","Data":"8b2de8dd59afd5c21c450798be89bf13788e2371f2ace7bf219d146b7d59487a"}
Sep 08 13:44:16.303133 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.303086    2580 scope.go:117] "RemoveContainer" containerID="2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"
Sep 08 13:44:16.312141 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.312124    2580 scope.go:117] "RemoveContainer" containerID="1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"
Sep 08 13:44:16.319627 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.319610    2580 scope.go:117] "RemoveContainer" containerID="e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"
Sep 08 13:44:16.326725 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.326707    2580 scope.go:117] "RemoveContainer" containerID="2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"
Sep 08 13:44:16.326965 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:44:16.326947    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d\": container with ID starting with 2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d not found: ID does not exist" containerID="2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"
Sep 08 13:44:16.327026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.326972    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d"} err="failed to get container status \"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d\": rpc error: code = NotFound desc = could not find container \"2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d\": container with ID starting with 2295fcc4a49a651a845434615dfb4251c9172d95d8f8c60038e4b681f8fd4c8d not found: ID does not exist"
Sep 08 13:44:16.327026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.326989    2580 scope.go:117] "RemoveContainer" containerID="1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"
Sep 08 13:44:16.327240 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:44:16.327222    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a\": container with ID starting with 1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a not found: ID does not exist" containerID="1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"
Sep 08 13:44:16.327293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.327247    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a"} err="failed to get container status \"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a\": rpc error: code = NotFound desc = could not find container \"1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a\": container with ID starting with 1caf312de7b68e79a5bfd0daf42b6a0360aa5dd8872b948ed78a7884c5c7089a not found: ID does not exist"
Sep 08 13:44:16.327293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.327270    2580 scope.go:117] "RemoveContainer" containerID="e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"
Sep 08 13:44:16.327499 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:44:16.327485    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763\": container with ID starting with e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763 not found: ID does not exist" containerID="e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"
Sep 08 13:44:16.327550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.327504    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763"} err="failed to get container status \"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763\": rpc error: code = NotFound desc = could not find container \"e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763\": container with ID starting with e49f2991a083cbdacf6ed1e07796881cd8e592f6ee7a717e2a0e0f51fe35d763 not found: ID does not exist"
Sep 08 13:44:16.331785 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.331759    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:44:16.337403 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.337382    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-qtkjw"]
Sep 08 13:44:16.536555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:16.536523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" path="/var/lib/kubelet/pods/cc9a4013-1007-4a98-b57b-24d79c682dce/volumes"
Sep 08 13:44:19.642792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.642755    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643103    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="storage-initializer"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="storage-initializer"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643121    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kserve-container"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kserve-container"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643136    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="storage-initializer"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="storage-initializer"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643154    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kube-rbac-proxy"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643159    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kube-rbac-proxy"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643184    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container"
Sep 08 13:44:19.643189 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643198    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kube-rbac-proxy"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643203    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kube-rbac-proxy"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643252    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kserve-container"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643259    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kserve-container"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643265    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d11d2ca-8154-4cfc-ac56-e81cda90f79d" containerName="kube-rbac-proxy"
Sep 08 13:44:19.643507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.643273    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9a4013-1007-4a98-b57b-24d79c682dce" containerName="kube-rbac-proxy"
Sep 08 13:44:19.647537 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.647520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.650535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.650512    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:44:19.650674 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.650655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:44:19.651530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.651508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 08 13:44:19.651664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.651601    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 08 13:44:19.651664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.651627    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:44:19.651664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.651572    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:44:19.651664 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.651568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:44:19.666719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.666694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:44:19.698199 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.698157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.698337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.698232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.698337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.698258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.698337 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.698281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxbn\" (UniqueName: \"kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.698452 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.698351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799224 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxbn\" (UniqueName: \"kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799708 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799883 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.799955 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.799936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.801743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.801722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.810916 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.810895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxbn\" (UniqueName: \"kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:19.958575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:19.958540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:20.250744 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:20.250721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:44:20.251943 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:44:20.251909    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68c79a1_0b61_4800_b753_5965bf322488.slice/crio-ce574dcee6d0f379efdf18d45ba175e20c3c77bee11e16eef35ba46920106b36 WatchSource:0}: Error finding container ce574dcee6d0f379efdf18d45ba175e20c3c77bee11e16eef35ba46920106b36: Status 404 returned error can't find the container with id ce574dcee6d0f379efdf18d45ba175e20c3c77bee11e16eef35ba46920106b36
Sep 08 13:44:20.323415 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:20.323385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerStarted","Data":"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"}
Sep 08 13:44:20.323565 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:20.323443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerStarted","Data":"ce574dcee6d0f379efdf18d45ba175e20c3c77bee11e16eef35ba46920106b36"}
Sep 08 13:44:29.367475 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:29.367444    2580 generic.go:356] "Generic (PLEG): container finished" podID="d68c79a1-0b61-4800-b753-5965bf322488" containerID="8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da" exitCode=0
Sep 08 13:44:29.367822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:29.367495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerDied","Data":"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"}
Sep 08 13:44:30.375919 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:30.375881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerStarted","Data":"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"}
Sep 08 13:44:30.376420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:30.375988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerStarted","Data":"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"}
Sep 08 13:44:30.376420 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:30.376214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:30.399311 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:30.399252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podStartSLOduration=2.3759834 podStartE2EDuration="11.399235696s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.023252296s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:30.398663891 +0000 UTC m=+1666.349317856" watchObservedRunningTime="2026-09-08 13:44:30.399235696 +0000 UTC m=+1666.349889648"
Sep 08 13:44:31.380260 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:31.380217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:31.381654 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:31.381625    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 08 13:44:32.384855 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:32.384818    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 08 13:44:37.389798 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:37.389766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:44:37.390355 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:37.390327    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 08 13:44:47.391355 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:44:47.391320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:45:01.852289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:01.852255    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:45:01.852748 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:01.852546    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container" containerID="cri-o://da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0" gracePeriod=30
Sep 08 13:45:01.852748 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:01.852621    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kube-rbac-proxy" containerID="cri-o://8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b" gracePeriod=30
Sep 08 13:45:02.385498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:02.385453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 08 13:45:02.538678 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:02.538641    2580 generic.go:356] "Generic (PLEG): container finished" podID="d68c79a1-0b61-4800-b753-5965bf322488" containerID="8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b" exitCode=2
Sep 08 13:45:02.538845 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:02.538683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerDied","Data":"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"}
Sep 08 13:45:03.500917 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.500895    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:45:03.546463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.546432    2580 generic.go:356] "Generic (PLEG): container finished" podID="d68c79a1-0b61-4800-b753-5965bf322488" containerID="da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0" exitCode=0
Sep 08 13:45:03.546618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.546487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"
Sep 08 13:45:03.546618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.546512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerDied","Data":"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"}
Sep 08 13:45:03.546618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.546580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49" event={"ID":"d68c79a1-0b61-4800-b753-5965bf322488","Type":"ContainerDied","Data":"ce574dcee6d0f379efdf18d45ba175e20c3c77bee11e16eef35ba46920106b36"}
Sep 08 13:45:03.546618 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.546617    2580 scope.go:117] "RemoveContainer" containerID="8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"
Sep 08 13:45:03.555578 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.555523    2580 scope.go:117] "RemoveContainer" containerID="da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"
Sep 08 13:45:03.565548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.565532    2580 scope.go:117] "RemoveContainer" containerID="8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"
Sep 08 13:45:03.573325 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.573299    2580 scope.go:117] "RemoveContainer" containerID="8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"
Sep 08 13:45:03.573620 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:03.573600    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b\": container with ID starting with 8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b not found: ID does not exist" containerID="8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"
Sep 08 13:45:03.573718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.573629    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b"} err="failed to get container status \"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b\": rpc error: code = NotFound desc = could not find container \"8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b\": container with ID starting with 8b6ad7a3fb157ea7b6268d4052cc88446de3e2d670051f1a6441fa47e650479b not found: ID does not exist"
Sep 08 13:45:03.573718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.573647    2580 scope.go:117] "RemoveContainer" containerID="da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"
Sep 08 13:45:03.573964 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:03.573941    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0\": container with ID starting with da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0 not found: ID does not exist" containerID="da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"
Sep 08 13:45:03.574057 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.573968    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0"} err="failed to get container status \"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0\": rpc error: code = NotFound desc = could not find container \"da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0\": container with ID starting with da64c240ca5d28671b0adc2e374a06a01520999b2bd1cf5c78ba3ed64d280ba0 not found: ID does not exist"
Sep 08 13:45:03.574057 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.573984    2580 scope.go:117] "RemoveContainer" containerID="8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"
Sep 08 13:45:03.574263 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:03.574246    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da\": container with ID starting with 8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da not found: ID does not exist" containerID="8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"
Sep 08 13:45:03.574315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.574276    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da"} err="failed to get container status \"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da\": rpc error: code = NotFound desc = could not find container \"8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da\": container with ID starting with 8f7fa048db8fbfc65b07a4775a4839f71a3e919ab8b7978a41bc7160d71758da not found: ID does not exist"
Sep 08 13:45:03.576533 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert\") pod \"d68c79a1-0b61-4800-b753-5965bf322488\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") "
Sep 08 13:45:03.576610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config\") pod \"d68c79a1-0b61-4800-b753-5965bf322488\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") "
Sep 08 13:45:03.576610 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location\") pod \"d68c79a1-0b61-4800-b753-5965bf322488\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") "
Sep 08 13:45:03.576703 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvxbn\" (UniqueName: \"kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn\") pod \"d68c79a1-0b61-4800-b753-5965bf322488\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") "
Sep 08 13:45:03.576762 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls\") pod \"d68c79a1-0b61-4800-b753-5965bf322488\" (UID: \"d68c79a1-0b61-4800-b753-5965bf322488\") "
Sep 08 13:45:03.576853 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d68c79a1-0b61-4800-b753-5965bf322488" (UID: "d68c79a1-0b61-4800-b753-5965bf322488"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:03.576949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.576930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d68c79a1-0b61-4800-b753-5965bf322488" (UID: "d68c79a1-0b61-4800-b753-5965bf322488"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:03.577159 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.577139    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:03.577319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.577298    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d68c79a1-0b61-4800-b753-5965bf322488-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:03.577623 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.577602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d68c79a1-0b61-4800-b753-5965bf322488" (UID: "d68c79a1-0b61-4800-b753-5965bf322488"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:03.578949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.578925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d68c79a1-0b61-4800-b753-5965bf322488" (UID: "d68c79a1-0b61-4800-b753-5965bf322488"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:03.579042 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.579022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn" (OuterVolumeSpecName: "kube-api-access-pvxbn") pod "d68c79a1-0b61-4800-b753-5965bf322488" (UID: "d68c79a1-0b61-4800-b753-5965bf322488"). InnerVolumeSpecName "kube-api-access-pvxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:03.678592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.678563    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d68c79a1-0b61-4800-b753-5965bf322488-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:03.678592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.678588    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d68c79a1-0b61-4800-b753-5965bf322488-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:03.678748 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.678599    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxbn\" (UniqueName: \"kubernetes.io/projected/d68c79a1-0b61-4800-b753-5965bf322488-kube-api-access-pvxbn\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:03.877007 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.876970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:45:03.881857 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:03.881813    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k49"]
Sep 08 13:45:04.535086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:04.535051    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68c79a1-0b61-4800-b753-5965bf322488" path="/var/lib/kubelet/pods/d68c79a1-0b61-4800-b753-5965bf322488/volumes"
Sep 08 13:45:07.204387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204302    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:07.204851 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204829    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container"
Sep 08 13:45:07.204928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204854    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container"
Sep 08 13:45:07.204928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204878    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="storage-initializer"
Sep 08 13:45:07.204928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204887    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="storage-initializer"
Sep 08 13:45:07.204928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204921    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kube-rbac-proxy"
Sep 08 13:45:07.205134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.204930    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kube-rbac-proxy"
Sep 08 13:45:07.205134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.205014    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kube-rbac-proxy"
Sep 08 13:45:07.205134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.205031    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68c79a1-0b61-4800-b753-5965bf322488" containerName="kserve-container"
Sep 08 13:45:07.210032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.210004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.213575 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.213553    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 08 13:45:07.213701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.213651    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:45:07.214855 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.214834    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 08 13:45:07.214988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.214854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:45:07.214988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.214907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:45:07.214988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.214926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:45:07.216315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.216296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:45:07.236091 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.236065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:07.310836 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.310803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.311025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.310846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.311025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.310890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.311025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.310963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhntj\" (UniqueName: \"kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.311025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.311003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412362 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhntj\" (UniqueName: \"kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.412814 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:07.412644    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 08 13:45:07.412814 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:07.412713    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls podName:05956008-2699-44ee-bb77-579a15cf044a nodeName:}" failed. No retries permitted until 2026-09-08 13:45:07.912694131 +0000 UTC m=+1703.863348074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" (UID: "05956008-2699-44ee-bb77-579a15cf044a") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 08 13:45:07.412949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.412848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.413096 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.413075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.413157 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.413112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.449792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.449763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhntj\" (UniqueName: \"kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.916053 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.916004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:07.918432 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:07.918408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:08.119714 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:08.119674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:08.403013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:08.402981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:08.406259 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:45:08.406234    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05956008_2699_44ee_bb77_579a15cf044a.slice/crio-974e526eb377346e787f7aa6a2917671042cc297ee154661bb2d718d104b0093 WatchSource:0}: Error finding container 974e526eb377346e787f7aa6a2917671042cc297ee154661bb2d718d104b0093: Status 404 returned error can't find the container with id 974e526eb377346e787f7aa6a2917671042cc297ee154661bb2d718d104b0093
Sep 08 13:45:08.571928 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:08.571886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerStarted","Data":"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"}
Sep 08 13:45:08.572149 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:08.571956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerStarted","Data":"974e526eb377346e787f7aa6a2917671042cc297ee154661bb2d718d104b0093"}
Sep 08 13:45:17.614926 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:17.614888    2580 generic.go:356] "Generic (PLEG): container finished" podID="05956008-2699-44ee-bb77-579a15cf044a" containerID="477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33" exitCode=0
Sep 08 13:45:17.615371 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:17.614964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerDied","Data":"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"}
Sep 08 13:45:18.621234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:18.621201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerStarted","Data":"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"}
Sep 08 13:45:18.621628 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:18.621264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerStarted","Data":"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"}
Sep 08 13:45:18.621628 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:18.621446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:18.646949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:18.646899    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podStartSLOduration=2.7233658800000002 podStartE2EDuration="11.6468839s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.92351802s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:18.645240525 +0000 UTC m=+1714.595894478" watchObservedRunningTime="2026-09-08 13:45:18.6468839 +0000 UTC m=+1714.597537851"
Sep 08 13:45:19.627359 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:19.627322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:19.628642 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:19.628611    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 08 13:45:20.632434 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:20.632391    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 08 13:45:25.636707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:25.636677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:25.637204 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:25.637124    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 08 13:45:35.638017 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:35.637987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:42.231140 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:42.231101    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:42.231598 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:42.231334    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container" containerID="cri-o://1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560" gracePeriod=30
Sep 08 13:45:42.231598 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:42.231414    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kube-rbac-proxy" containerID="cri-o://a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e" gracePeriod=30
Sep 08 13:45:42.740300 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:42.740267    2580 generic.go:356] "Generic (PLEG): container finished" podID="05956008-2699-44ee-bb77-579a15cf044a" containerID="a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e" exitCode=2
Sep 08 13:45:42.740300 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:42.740306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerDied","Data":"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"}
Sep 08 13:45:43.672627 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.672595    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:43.731894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.731867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert\") pod \"05956008-2699-44ee-bb77-579a15cf044a\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") "
Sep 08 13:45:43.732086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.731943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location\") pod \"05956008-2699-44ee-bb77-579a15cf044a\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") "
Sep 08 13:45:43.732086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.731989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhntj\" (UniqueName: \"kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj\") pod \"05956008-2699-44ee-bb77-579a15cf044a\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") "
Sep 08 13:45:43.732086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.732022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config\") pod \"05956008-2699-44ee-bb77-579a15cf044a\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") "
Sep 08 13:45:43.732086 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.732055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") pod \"05956008-2699-44ee-bb77-579a15cf044a\" (UID: \"05956008-2699-44ee-bb77-579a15cf044a\") "
Sep 08 13:45:43.732316 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.732221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05956008-2699-44ee-bb77-579a15cf044a" (UID: "05956008-2699-44ee-bb77-579a15cf044a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:43.732367 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.732329    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:43.732464 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.732432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05956008-2699-44ee-bb77-579a15cf044a" (UID: "05956008-2699-44ee-bb77-579a15cf044a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:43.733349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.733315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05956008-2699-44ee-bb77-579a15cf044a" (UID: "05956008-2699-44ee-bb77-579a15cf044a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:43.734033 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.734007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05956008-2699-44ee-bb77-579a15cf044a" (UID: "05956008-2699-44ee-bb77-579a15cf044a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:43.734106 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.734071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj" (OuterVolumeSpecName: "kube-api-access-zhntj") pod "05956008-2699-44ee-bb77-579a15cf044a" (UID: "05956008-2699-44ee-bb77-579a15cf044a"). InnerVolumeSpecName "kube-api-access-zhntj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:43.747436 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.747407    2580 generic.go:356] "Generic (PLEG): container finished" podID="05956008-2699-44ee-bb77-579a15cf044a" containerID="1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560" exitCode=0
Sep 08 13:45:43.747539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.747460    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"
Sep 08 13:45:43.747539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.747467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerDied","Data":"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"}
Sep 08 13:45:43.747539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.747514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6" event={"ID":"05956008-2699-44ee-bb77-579a15cf044a","Type":"ContainerDied","Data":"974e526eb377346e787f7aa6a2917671042cc297ee154661bb2d718d104b0093"}
Sep 08 13:45:43.747539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.747539    2580 scope.go:117] "RemoveContainer" containerID="a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"
Sep 08 13:45:43.756644 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.756629    2580 scope.go:117] "RemoveContainer" containerID="1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"
Sep 08 13:45:43.764006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.763986    2580 scope.go:117] "RemoveContainer" containerID="477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"
Sep 08 13:45:43.771358 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.771334    2580 scope.go:117] "RemoveContainer" containerID="a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"
Sep 08 13:45:43.771625 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:43.771605    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e\": container with ID starting with a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e not found: ID does not exist" containerID="a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"
Sep 08 13:45:43.771707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.771647    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e"} err="failed to get container status \"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e\": rpc error: code = NotFound desc = could not find container \"a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e\": container with ID starting with a2328e4a4cb112984525f2cdc316fedd28f7ad6761d75e9f0d7eda83acd5c25e not found: ID does not exist"
Sep 08 13:45:43.771707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.771674    2580 scope.go:117] "RemoveContainer" containerID="1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"
Sep 08 13:45:43.771921 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:43.771905    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560\": container with ID starting with 1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560 not found: ID does not exist" containerID="1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"
Sep 08 13:45:43.771987 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.771928    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560"} err="failed to get container status \"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560\": rpc error: code = NotFound desc = could not find container \"1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560\": container with ID starting with 1da00d8385421b83f46f4c25c131c7ddd3e4b2bd1e6d479979f36685c3cee560 not found: ID does not exist"
Sep 08 13:45:43.771987 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.771959    2580 scope.go:117] "RemoveContainer" containerID="477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"
Sep 08 13:45:43.772257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.772238    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:43.772322 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:43.772276    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33\": container with ID starting with 477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33 not found: ID does not exist" containerID="477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"
Sep 08 13:45:43.772322 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.772299    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33"} err="failed to get container status \"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33\": rpc error: code = NotFound desc = could not find container \"477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33\": container with ID starting with 477c8026e2a2d6fdbc33a9cf0260ba4235dcadd7f1c8efae635e8c79e08cea33 not found: ID does not exist"
Sep 08 13:45:43.778991 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.778966    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-gb4q6"]
Sep 08 13:45:43.833218 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.833155    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhntj\" (UniqueName: \"kubernetes.io/projected/05956008-2699-44ee-bb77-579a15cf044a-kube-api-access-zhntj\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:43.833218 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.833216    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05956008-2699-44ee-bb77-579a15cf044a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:43.833397 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.833227    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05956008-2699-44ee-bb77-579a15cf044a-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:43.833397 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:43.833236    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05956008-2699-44ee-bb77-579a15cf044a-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:45:44.536136 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:44.536097    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05956008-2699-44ee-bb77-579a15cf044a" path="/var/lib/kubelet/pods/05956008-2699-44ee-bb77-579a15cf044a/volumes"
Sep 08 13:45:47.622374 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622338    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622671    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="storage-initializer"
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622683    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="storage-initializer"
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container"
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622708    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container"
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622715    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kube-rbac-proxy"
Sep 08 13:45:47.622764 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kube-rbac-proxy"
Sep 08 13:45:47.622988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622776    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kserve-container"
Sep 08 13:45:47.622988 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.622785    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05956008-2699-44ee-bb77-579a15cf044a" containerName="kube-rbac-proxy"
Sep 08 13:45:47.627547 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.627526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.635228 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.635200    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 08 13:45:47.635353 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.635201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 08 13:45:47.635353 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.635263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:45:47.637702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.637687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:45:47.637778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.637762    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:45:47.644639 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.638866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:45:47.644639 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.640856    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:45:47.656120 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.656090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:45:47.666247 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.666218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8dg\" (UniqueName: \"kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.666388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.666265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.666388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.666289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.666479 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.666425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.666525 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.666504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.749593 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.749558    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:45:47.753046 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.753029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.759372 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.759348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 08 13:45:47.767455 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.767572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8dg\" (UniqueName: \"kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.767572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.767572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.767572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.767718 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:47.767642    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 08 13:45:47.767718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.767718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.767718 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:47.767712    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls podName:27908ab9-7f61-43da-8b88-d4453c705eaa nodeName:}" failed. No retries permitted until 2026-09-08 13:45:48.267692818 +0000 UTC m=+1744.218346750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls") pod "isvc-canary-create-v2-predictor-7659888cb8-8jtwm" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 08 13:45:47.767894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kps\" (UniqueName: \"kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.767894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.767894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.768019 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.767921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.768251 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.768234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.768352 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.768337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.781139 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.781112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:45:47.795797 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.795738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8dg\" (UniqueName: \"kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:47.868368 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kps\" (UniqueName: \"kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.868368 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.868564 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.868564 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.868564 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.868682 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:47.868579    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 08 13:45:47.868682 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:45:47.868650    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls podName:18bb23d9-fc18-4fdb-a988-d576881565fb nodeName:}" failed. No retries permitted until 2026-09-08 13:45:48.368629326 +0000 UTC m=+1744.319283271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls") pod "isvc-canary-create-predictor-8488566dd5-9bqpf" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 08 13:45:47.868871 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.868848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.869121 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.869105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.869158 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.869132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:47.895254 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:47.895199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kps\" (UniqueName: \"kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:48.270939 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.270890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:48.273306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.273281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-8jtwm\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:48.372239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.372207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:48.374437 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.374415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-9bqpf\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:48.537925 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.537852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:48.662339 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.662305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:48.829239 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.829215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:45:48.831661 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:45:48.831630    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27908ab9_7f61_43da_8b88_d4453c705eaa.slice/crio-49344a7497c9b0382b6986310620f933a46dbcd61a9e6d7757a4c4eee0f1e345 WatchSource:0}: Error finding container 49344a7497c9b0382b6986310620f933a46dbcd61a9e6d7757a4c4eee0f1e345: Status 404 returned error can't find the container with id 49344a7497c9b0382b6986310620f933a46dbcd61a9e6d7757a4c4eee0f1e345
Sep 08 13:45:48.953044 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:48.953008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:45:48.956831 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:45:48.956801    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb23d9_fc18_4fdb_a988_d576881565fb.slice/crio-7705eb97d88f30d02b30e6dfd5b0e6b5db4dd8eb4276b18c118d2d2d5f4b17d6 WatchSource:0}: Error finding container 7705eb97d88f30d02b30e6dfd5b0e6b5db4dd8eb4276b18c118d2d2d5f4b17d6: Status 404 returned error can't find the container with id 7705eb97d88f30d02b30e6dfd5b0e6b5db4dd8eb4276b18c118d2d2d5f4b17d6
Sep 08 13:45:49.776658 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:49.776623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerStarted","Data":"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"}
Sep 08 13:45:49.777059 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:49.776683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerStarted","Data":"49344a7497c9b0382b6986310620f933a46dbcd61a9e6d7757a4c4eee0f1e345"}
Sep 08 13:45:49.778220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:49.778194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerStarted","Data":"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"}
Sep 08 13:45:49.778317 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:49.778243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerStarted","Data":"7705eb97d88f30d02b30e6dfd5b0e6b5db4dd8eb4276b18c118d2d2d5f4b17d6"}
Sep 08 13:45:53.799446 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:53.799413    2580 generic.go:356] "Generic (PLEG): container finished" podID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerID="dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f" exitCode=0
Sep 08 13:45:53.799872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:53.799495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerDied","Data":"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"}
Sep 08 13:45:53.801223 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:53.801197    2580 generic.go:356] "Generic (PLEG): container finished" podID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerID="18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d" exitCode=0
Sep 08 13:45:53.801331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:53.801244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerDied","Data":"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"}
Sep 08 13:45:54.807791 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.807757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerStarted","Data":"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"}
Sep 08 13:45:54.808275 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.807832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerStarted","Data":"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"}
Sep 08 13:45:54.808275 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.808116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:54.808275 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.808185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:45:54.810056 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.810030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:45:54.810215 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.810191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerStarted","Data":"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"}
Sep 08 13:45:54.810279 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.810254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerStarted","Data":"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"}
Sep 08 13:45:54.810510 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.810492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:54.810549 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.810535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:45:54.811638 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.811613    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:45:54.834702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.834659    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podStartSLOduration=3.838115267 podStartE2EDuration="7.834647561s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996532294s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:54.834386469 +0000 UTC m=+1750.785040422" watchObservedRunningTime="2026-09-08 13:45:54.834647561 +0000 UTC m=+1750.785301513"
Sep 08 13:45:54.866818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:54.866760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podStartSLOduration=3.879999102 podStartE2EDuration="7.866740589s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986741487s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:54.8644636 +0000 UTC m=+1750.815117553" watchObservedRunningTime="2026-09-08 13:45:54.866740589 +0000 UTC m=+1750.817394542"
Sep 08 13:45:55.289522 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.289485    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:45:55.292817 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.292801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.296132 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.296107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 08 13:45:55.296268 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.296175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:45:55.309421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.309394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:45:55.336208 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.336159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.336343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.336214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcqd\" (UniqueName: \"kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.336343 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.336296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.336458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.336416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.336505 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.336479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437314 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcqd\" (UniqueName: \"kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437314 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437451 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437451 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.437806 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.438000 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.437981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.438051 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.438001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.439828 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.439804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.446983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.446966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcqd\" (UniqueName: \"kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd\") pod \"isvc-sklearn-runtime-predictor-989d67dd8-t4n2s\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.603404 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.603308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:45:55.815849 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.815804    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:45:55.816336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.815932    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:45:55.877916 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:55.877862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:45:55.880099 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:45:55.880073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1c2587_5d83_40b3_be9b_ecac1f5fc7f0.slice/crio-af187afcdee4895c27400ccf2ee85a3344453f821d79a02ccc56170ab3d208fd WatchSource:0}: Error finding container af187afcdee4895c27400ccf2ee85a3344453f821d79a02ccc56170ab3d208fd: Status 404 returned error can't find the container with id af187afcdee4895c27400ccf2ee85a3344453f821d79a02ccc56170ab3d208fd
Sep 08 13:45:56.822368 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:56.822328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerStarted","Data":"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"}
Sep 08 13:45:56.822818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:45:56.822387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerStarted","Data":"af187afcdee4895c27400ccf2ee85a3344453f821d79a02ccc56170ab3d208fd"}
Sep 08 13:46:00.820603 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:00.820571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:46:00.821010 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:00.820892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:46:00.821283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:00.821095    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:46:00.821390 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:00.821359    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:01.854964 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:01.854930    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerID="d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98" exitCode=0
Sep 08 13:46:01.855344 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:01.855002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerDied","Data":"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"}
Sep 08 13:46:02.862707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.862672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerStarted","Data":"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"}
Sep 08 13:46:02.863107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.862735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerStarted","Data":"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"}
Sep 08 13:46:02.863107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.862952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:02.863107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.863013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:02.864598 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.864570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 08 13:46:02.889645 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:02.889593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podStartSLOduration=2.069277045 podStartE2EDuration="7.889578116s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.820301071s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:02.887339512 +0000 UTC m=+1758.837993463" watchObservedRunningTime="2026-09-08 13:46:02.889578116 +0000 UTC m=+1758.840232068"
Sep 08 13:46:03.870148 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:03.870099    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 08 13:46:08.875859 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:08.875829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:08.876447 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:08.876412    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 08 13:46:10.821909 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:10.821860    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:10.822421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:10.822392    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:46:18.877259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:18.877225    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:20.821709 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:20.821665    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:20.822094 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:20.821665    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:46:27.198800 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:27.198764    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:46:27.199228 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:27.198999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container" containerID="cri-o://9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220" gracePeriod=30
Sep 08 13:46:27.199228 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:27.199050    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kube-rbac-proxy" containerID="cri-o://0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7" gracePeriod=30
Sep 08 13:46:28.007501 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.007468    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerID="0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7" exitCode=2
Sep 08 13:46:28.007755 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.007543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerDied","Data":"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"}
Sep 08 13:46:28.248386 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.248362    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:28.331822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.331736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert\") pod \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") "
Sep 08 13:46:28.331822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.331797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls\") pod \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") "
Sep 08 13:46:28.332014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.331827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location\") pod \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") "
Sep 08 13:46:28.332014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.331873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcqd\" (UniqueName: \"kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd\") pod \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") "
Sep 08 13:46:28.332014 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.331911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config\") pod \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\" (UID: \"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0\") "
Sep 08 13:46:28.332203 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.332136    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" (UID: "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:28.332351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.332327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" (UID: "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:28.333985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.333962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" (UID: "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:46:28.334083 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.334020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd" (OuterVolumeSpecName: "kube-api-access-mzcqd") pod "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" (UID: "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0"). InnerVolumeSpecName "kube-api-access-mzcqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:28.356634 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.356608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" (UID: "0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:28.432823 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.432794    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcqd\" (UniqueName: \"kubernetes.io/projected/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-api-access-mzcqd\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:46:28.432823 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.432821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:46:28.432967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.432831    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:46:28.432967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.432841    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:46:28.432967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:28.432849    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:46:29.014377 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.014341    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerID="9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220" exitCode=0
Sep 08 13:46:29.014550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.014407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"
Sep 08 13:46:29.014550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.014428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerDied","Data":"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"}
Sep 08 13:46:29.014550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.014490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s" event={"ID":"0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0","Type":"ContainerDied","Data":"af187afcdee4895c27400ccf2ee85a3344453f821d79a02ccc56170ab3d208fd"}
Sep 08 13:46:29.014550 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.014517    2580 scope.go:117] "RemoveContainer" containerID="0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"
Sep 08 13:46:29.022945 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.022926    2580 scope.go:117] "RemoveContainer" containerID="9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"
Sep 08 13:46:29.030723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.030701    2580 scope.go:117] "RemoveContainer" containerID="d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"
Sep 08 13:46:29.038044 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038028    2580 scope.go:117] "RemoveContainer" containerID="0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"
Sep 08 13:46:29.038327 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:46:29.038307    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7\": container with ID starting with 0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7 not found: ID does not exist" containerID="0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"
Sep 08 13:46:29.038383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038334    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7"} err="failed to get container status \"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7\": rpc error: code = NotFound desc = could not find container \"0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7\": container with ID starting with 0b7a60eddee768a5d57c8e9bb2b61bfbc350e90b9ab70a1f5efa9d9e3a3adfa7 not found: ID does not exist"
Sep 08 13:46:29.038383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038349    2580 scope.go:117] "RemoveContainer" containerID="9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"
Sep 08 13:46:29.038556 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:46:29.038540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220\": container with ID starting with 9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220 not found: ID does not exist" containerID="9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"
Sep 08 13:46:29.038599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038560    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220"} err="failed to get container status \"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220\": rpc error: code = NotFound desc = could not find container \"9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220\": container with ID starting with 9a46d3b0ecbad9766d21da61be7ac5195f0ecb681ff178f3796c74be38014220 not found: ID does not exist"
Sep 08 13:46:29.038599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038573    2580 scope.go:117] "RemoveContainer" containerID="d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"
Sep 08 13:46:29.038780 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:46:29.038762    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98\": container with ID starting with d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98 not found: ID does not exist" containerID="d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"
Sep 08 13:46:29.038820 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.038784    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98"} err="failed to get container status \"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98\": rpc error: code = NotFound desc = could not find container \"d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98\": container with ID starting with d34dfa44590f0595115f165f62845c197ab6e358674590819a000105f5eadb98 not found: ID does not exist"
Sep 08 13:46:29.045204 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.045155    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:46:29.047606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:29.047578    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-989d67dd8-t4n2s"]
Sep 08 13:46:30.535971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:30.535936    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" path="/var/lib/kubelet/pods/0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0/volumes"
Sep 08 13:46:30.821555 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:30.821466    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:30.821704 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:30.821477    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:46:40.822142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:40.822061    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:46:40.822589 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:40.822062    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:50.821585 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:50.821546    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:46:50.822010 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:46:50.821543    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:47:00.821785 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:00.821746    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:47:00.822259 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:00.821747    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:47:10.821945 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:10.821916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:47:10.822349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:10.822000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:47:17.692730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.692685    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:47:17.693323 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.693010    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" containerID="cri-o://e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3" gracePeriod=30
Sep 08 13:47:17.693323 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.693031    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kube-rbac-proxy" containerID="cri-o://11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9" gracePeriod=30
Sep 08 13:47:17.699675 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.699651    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:47:17.699889 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.699867    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" containerID="cri-o://a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172" gracePeriod=30
Sep 08 13:47:17.700006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:17.699960    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kube-rbac-proxy" containerID="cri-o://ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6" gracePeriod=30
Sep 08 13:47:18.259539 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:18.259503    2580 generic.go:356] "Generic (PLEG): container finished" podID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerID="ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6" exitCode=2
Sep 08 13:47:18.259737 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:18.259571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerDied","Data":"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"}
Sep 08 13:47:18.261557 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:18.261535    2580 generic.go:356] "Generic (PLEG): container finished" podID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerID="11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9" exitCode=2
Sep 08 13:47:18.261675 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:18.261589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerDied","Data":"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"}
Sep 08 13:47:19.078104 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:19.078022    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:47:19.080702 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:19.080682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:47:20.816047 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:20.815998    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 08 13:47:20.816465 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:20.816013    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 08 13:47:20.821345 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:20.821325    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 08 13:47:20.821345 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:20.821333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 08 13:47:22.643201 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.643160    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:47:22.694361 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert\") pod \"18bb23d9-fc18-4fdb-a988-d576881565fb\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") "
Sep 08 13:47:22.694543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5kps\" (UniqueName: \"kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps\") pod \"18bb23d9-fc18-4fdb-a988-d576881565fb\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") "
Sep 08 13:47:22.694543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694497    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location\") pod \"18bb23d9-fc18-4fdb-a988-d576881565fb\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") "
Sep 08 13:47:22.694543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config\") pod \"18bb23d9-fc18-4fdb-a988-d576881565fb\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") "
Sep 08 13:47:22.694730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") pod \"18bb23d9-fc18-4fdb-a988-d576881565fb\" (UID: \"18bb23d9-fc18-4fdb-a988-d576881565fb\") "
Sep 08 13:47:22.694815 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18bb23d9-fc18-4fdb-a988-d576881565fb" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:22.694879 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18bb23d9-fc18-4fdb-a988-d576881565fb" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:22.694939 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.694892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18bb23d9-fc18-4fdb-a988-d576881565fb" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:22.696592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.696562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps" (OuterVolumeSpecName: "kube-api-access-j5kps") pod "18bb23d9-fc18-4fdb-a988-d576881565fb" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb"). InnerVolumeSpecName "kube-api-access-j5kps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:22.696707 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.696685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18bb23d9-fc18-4fdb-a988-d576881565fb" (UID: "18bb23d9-fc18-4fdb-a988-d576881565fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:22.744957 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.744930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:47:22.795487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config\") pod \"27908ab9-7f61-43da-8b88-d4453c705eaa\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") "
Sep 08 13:47:22.795669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") pod \"27908ab9-7f61-43da-8b88-d4453c705eaa\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") "
Sep 08 13:47:22.795669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert\") pod \"27908ab9-7f61-43da-8b88-d4453c705eaa\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") "
Sep 08 13:47:22.795669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8dg\" (UniqueName: \"kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg\") pod \"27908ab9-7f61-43da-8b88-d4453c705eaa\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") "
Sep 08 13:47:22.795669 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location\") pod \"27908ab9-7f61-43da-8b88-d4453c705eaa\" (UID: \"27908ab9-7f61-43da-8b88-d4453c705eaa\") "
Sep 08 13:47:22.795873 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.795812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27908ab9-7f61-43da-8b88-d4453c705eaa" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:22.796055 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27908ab9-7f61-43da-8b88-d4453c705eaa" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27908ab9-7f61-43da-8b88-d4453c705eaa" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796041    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796083    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796096    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5kps\" (UniqueName: \"kubernetes.io/projected/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-api-access-j5kps\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796105    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18bb23d9-fc18-4fdb-a988-d576881565fb-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796114    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18bb23d9-fc18-4fdb-a988-d576881565fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.796130 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.796128    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18bb23d9-fc18-4fdb-a988-d576881565fb-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.797638 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.797613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27908ab9-7f61-43da-8b88-d4453c705eaa" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:22.797736 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.797656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg" (OuterVolumeSpecName: "kube-api-access-8j8dg") pod "27908ab9-7f61-43da-8b88-d4453c705eaa" (UID: "27908ab9-7f61-43da-8b88-d4453c705eaa"). InnerVolumeSpecName "kube-api-access-8j8dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:22.854427 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854398    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:47:22.854728 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854745    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="storage-initializer"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854760    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="storage-initializer"
Sep 08 13:47:22.854769 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854766    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="storage-initializer"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854771    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="storage-initializer"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854802    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854808    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="storage-initializer"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854814    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="storage-initializer"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854823    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854891    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854898    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854903    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854909    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerName="kube-rbac-proxy"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854916    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kserve-container"
Sep 08 13:47:22.854950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.854924    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e1c2587-5d83-40b3-be9b-ecac1f5fc7f0" containerName="kube-rbac-proxy"
Sep 08 13:47:22.858245 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.858229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.860970 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.860954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 08 13:47:22.861242 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.861223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:47:22.868820 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.868799    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:47:22.872069 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.872053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.876198 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.876179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 08 13:47:22.879488 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.879464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:47:22.897336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.897438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.897438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.897438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.897438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.897438 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.897630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.897630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh22p\" (UniqueName: \"kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.897630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dxf\" (UniqueName: \"kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.897630 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.897777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897679    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27908ab9-7f61-43da-8b88-d4453c705eaa-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.897777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897694    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27908ab9-7f61-43da-8b88-d4453c705eaa-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.897777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897703    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27908ab9-7f61-43da-8b88-d4453c705eaa-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.897777 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.897712    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8dg\" (UniqueName: \"kubernetes.io/projected/27908ab9-7f61-43da-8b88-d4453c705eaa-kube-api-access-8j8dg\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:47:22.905410 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.905387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:47:22.998679 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh22p\" (UniqueName: \"kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.998679 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dxf\" (UniqueName: \"kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.998679 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.998971 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.998867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.999328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.999328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.999328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.999644 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:22.999745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.999811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:22.999950 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:22.999930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:23.001356 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.001340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:23.001436 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.001419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:23.011063 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.011037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dxf\" (UniqueName: \"kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf\") pod \"isvc-canary-promote-v2-predictor-67454bd898-httr7\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:23.012090 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.012067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh22p\" (UniqueName: \"kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p\") pod \"isvc-canary-promote-predictor-64849fbd89-j2tb9\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:23.169374 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.169329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:23.181735 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.181712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:23.287800 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.287727    2580 generic.go:356] "Generic (PLEG): container finished" podID="18bb23d9-fc18-4fdb-a988-d576881565fb" containerID="e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3" exitCode=0
Sep 08 13:47:23.287800 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.287783    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"
Sep 08 13:47:23.287967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.287804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerDied","Data":"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"}
Sep 08 13:47:23.287967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.287874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf" event={"ID":"18bb23d9-fc18-4fdb-a988-d576881565fb","Type":"ContainerDied","Data":"7705eb97d88f30d02b30e6dfd5b0e6b5db4dd8eb4276b18c118d2d2d5f4b17d6"}
Sep 08 13:47:23.287967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.287914    2580 scope.go:117] "RemoveContainer" containerID="11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"
Sep 08 13:47:23.290069 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.290047    2580 generic.go:356] "Generic (PLEG): container finished" podID="27908ab9-7f61-43da-8b88-d4453c705eaa" containerID="a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172" exitCode=0
Sep 08 13:47:23.290255 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.290076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerDied","Data":"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"}
Sep 08 13:47:23.290255 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.290096    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"
Sep 08 13:47:23.290255 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.290113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm" event={"ID":"27908ab9-7f61-43da-8b88-d4453c705eaa","Type":"ContainerDied","Data":"49344a7497c9b0382b6986310620f933a46dbcd61a9e6d7757a4c4eee0f1e345"}
Sep 08 13:47:23.297780 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.297733    2580 scope.go:117] "RemoveContainer" containerID="e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"
Sep 08 13:47:23.305592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.305576    2580 scope.go:117] "RemoveContainer" containerID="18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"
Sep 08 13:47:23.312827 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.312810    2580 scope.go:117] "RemoveContainer" containerID="11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"
Sep 08 13:47:23.313097 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.313077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9\": container with ID starting with 11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9 not found: ID does not exist" containerID="11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"
Sep 08 13:47:23.313144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313106    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9"} err="failed to get container status \"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9\": rpc error: code = NotFound desc = could not find container \"11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9\": container with ID starting with 11f046c0ab2ab4216652ac2f111eb3d2c7b0e39d87d8d7d644dbbd10f4de11d9 not found: ID does not exist"
Sep 08 13:47:23.313144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313123    2580 scope.go:117] "RemoveContainer" containerID="e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"
Sep 08 13:47:23.313390 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.313374    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3\": container with ID starting with e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3 not found: ID does not exist" containerID="e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"
Sep 08 13:47:23.313435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313394    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3"} err="failed to get container status \"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3\": rpc error: code = NotFound desc = could not find container \"e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3\": container with ID starting with e68b550bcfb86c13517a81da98bb5331c606bdf3a5d4bbd2804793941d2a10e3 not found: ID does not exist"
Sep 08 13:47:23.313435 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313409    2580 scope.go:117] "RemoveContainer" containerID="18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"
Sep 08 13:47:23.313624 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.313612    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d\": container with ID starting with 18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d not found: ID does not exist" containerID="18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"
Sep 08 13:47:23.313673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d"} err="failed to get container status \"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d\": rpc error: code = NotFound desc = could not find container \"18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d\": container with ID starting with 18e493b0eff823afc862041e4c050fecf0fc447578e02f26bd1c07b511ee4d1d not found: ID does not exist"
Sep 08 13:47:23.313673 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.313643    2580 scope.go:117] "RemoveContainer" containerID="ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"
Sep 08 13:47:23.316142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.316120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:47:23.321204 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.321184    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-8jtwm"]
Sep 08 13:47:23.321457 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.321441    2580 scope.go:117] "RemoveContainer" containerID="a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"
Sep 08 13:47:23.328807 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.328790    2580 scope.go:117] "RemoveContainer" containerID="dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"
Sep 08 13:47:23.332912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.332888    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:47:23.336313 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.336297    2580 scope.go:117] "RemoveContainer" containerID="ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"
Sep 08 13:47:23.336575 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.336557    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6\": container with ID starting with ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6 not found: ID does not exist" containerID="ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"
Sep 08 13:47:23.336650 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.336581    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6"} err="failed to get container status \"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6\": rpc error: code = NotFound desc = could not find container \"ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6\": container with ID starting with ee14d43a025029e61497bc2ec720d991b520056f40d3dd70c46eb810b7422db6 not found: ID does not exist"
Sep 08 13:47:23.336650 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.336601    2580 scope.go:117] "RemoveContainer" containerID="a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"
Sep 08 13:47:23.337007 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.336985    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172\": container with ID starting with a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172 not found: ID does not exist" containerID="a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"
Sep 08 13:47:23.337146 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.337014    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172"} err="failed to get container status \"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172\": rpc error: code = NotFound desc = could not find container \"a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172\": container with ID starting with a3f3bccfba425c0194c3a21677d749f320199d75de924db08e3bc06f88e22172 not found: ID does not exist"
Sep 08 13:47:23.337146 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.337032    2580 scope.go:117] "RemoveContainer" containerID="dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"
Sep 08 13:47:23.337496 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:47:23.337476    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f\": container with ID starting with dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f not found: ID does not exist" containerID="dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"
Sep 08 13:47:23.337612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.337496    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f"} err="failed to get container status \"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f\": rpc error: code = NotFound desc = could not find container \"dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f\": container with ID starting with dfb5e0e0cdbe7b69e4fe4c3e389c2d0e28b73cbd29d292e7925793d8294abb0f not found: ID does not exist"
Sep 08 13:47:23.339078 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.339056    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-9bqpf"]
Sep 08 13:47:23.448772 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.448748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:47:23.450736 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:47:23.450698    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b525b7_94c4_4dd8_80c0_6f3460dee60f.slice/crio-18fe1806171041d8b225c471f50614294f7f7d17ac092db7f1e7a086f8460df3 WatchSource:0}: Error finding container 18fe1806171041d8b225c471f50614294f7f7d17ac092db7f1e7a086f8460df3: Status 404 returned error can't find the container with id 18fe1806171041d8b225c471f50614294f7f7d17ac092db7f1e7a086f8460df3
Sep 08 13:47:23.464999 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:23.464974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:47:23.469262 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:47:23.469239    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cccc1d_bf8d_4a54_aa2a_0c571581da99.slice/crio-6e9df0c5582b66b8bab862a92098d722148121b1f6a9e68c642287b9d891a7fd WatchSource:0}: Error finding container 6e9df0c5582b66b8bab862a92098d722148121b1f6a9e68c642287b9d891a7fd: Status 404 returned error can't find the container with id 6e9df0c5582b66b8bab862a92098d722148121b1f6a9e68c642287b9d891a7fd
Sep 08 13:47:24.298123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.298085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerStarted","Data":"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"}
Sep 08 13:47:24.298533 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.298183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerStarted","Data":"6e9df0c5582b66b8bab862a92098d722148121b1f6a9e68c642287b9d891a7fd"}
Sep 08 13:47:24.299535 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.299506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerStarted","Data":"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"}
Sep 08 13:47:24.299652 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.299568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerStarted","Data":"18fe1806171041d8b225c471f50614294f7f7d17ac092db7f1e7a086f8460df3"}
Sep 08 13:47:24.536370 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.536336    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bb23d9-fc18-4fdb-a988-d576881565fb" path="/var/lib/kubelet/pods/18bb23d9-fc18-4fdb-a988-d576881565fb/volumes"
Sep 08 13:47:24.536830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:24.536818    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27908ab9-7f61-43da-8b88-d4453c705eaa" path="/var/lib/kubelet/pods/27908ab9-7f61-43da-8b88-d4453c705eaa/volumes"
Sep 08 13:47:28.320461 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:28.320423    2580 generic.go:356] "Generic (PLEG): container finished" podID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerID="1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358" exitCode=0
Sep 08 13:47:28.320936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:28.320496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerDied","Data":"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"}
Sep 08 13:47:28.322113 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:28.322090    2580 generic.go:356] "Generic (PLEG): container finished" podID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerID="2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098" exitCode=0
Sep 08 13:47:28.322231 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:28.322159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerDied","Data":"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"}
Sep 08 13:47:29.328895 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.328853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerStarted","Data":"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"}
Sep 08 13:47:29.329349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.328927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerStarted","Data":"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"}
Sep 08 13:47:29.329349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.329183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:29.329349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.329342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:29.330933 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.330907    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:47:29.331211 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.331193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerStarted","Data":"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"}
Sep 08 13:47:29.331303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.331250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerStarted","Data":"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"}
Sep 08 13:47:29.331470 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.331452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:29.351104 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.351066    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podStartSLOduration=3.330722976 podStartE2EDuration="7.351051957s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.020328981s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:29.350020175 +0000 UTC m=+1845.300674118" watchObservedRunningTime="2026-09-08 13:47:29.351051957 +0000 UTC m=+1845.301705909"
Sep 08 13:47:29.372330 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:29.372293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podStartSLOduration=3.359018229 podStartE2EDuration="7.372280795s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.013262566s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:29.370841807 +0000 UTC m=+1845.321495784" watchObservedRunningTime="2026-09-08 13:47:29.372280795 +0000 UTC m=+1845.322934746"
Sep 08 13:47:30.337793 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:30.337754    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:47:30.338152 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:30.337843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:30.339112 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:30.339085    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:47:31.342447 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:31.342411    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:47:35.342481 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:35.342448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:47:35.342929 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:35.342902    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:47:36.347032 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:36.347004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:47:36.347600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:36.347564    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:47:37.749648 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.749610    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:47:37.753567 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.753550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.757001 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.756979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 08 13:47:37.757116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.757069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 08 13:47:37.770860 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.770833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:47:37.830894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.830866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr669\" (UniqueName: \"kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.831064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.830916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.831064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.830952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.831064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.831024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.831249 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.831075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932448 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr669\" (UniqueName: \"kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.932949 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.932928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.933261 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.933237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.933309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.933237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.935006 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.934988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:37.945743 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:37.945718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr669\" (UniqueName: \"kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669\") pod \"isvc-sklearn-v2-predictor-687875ffcb-2g8rl\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:38.063681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:38.063588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:38.347245 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:38.347152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:47:38.350602 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:47:38.350576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ad2997_d3c4_4c77_9e11_228f9d525e64.slice/crio-a00997caf9e8e65202ba486a5610b334e2f535b9f4f63f9d93d0ba8f0909b400 WatchSource:0}: Error finding container a00997caf9e8e65202ba486a5610b334e2f535b9f4f63f9d93d0ba8f0909b400: Status 404 returned error can't find the container with id a00997caf9e8e65202ba486a5610b334e2f535b9f4f63f9d93d0ba8f0909b400
Sep 08 13:47:38.378026 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:38.378000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerStarted","Data":"a00997caf9e8e65202ba486a5610b334e2f535b9f4f63f9d93d0ba8f0909b400"}
Sep 08 13:47:39.385119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:39.385080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerStarted","Data":"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"}
Sep 08 13:47:43.406880 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:43.406842    2580 generic.go:356] "Generic (PLEG): container finished" podID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerID="c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956" exitCode=0
Sep 08 13:47:43.407362 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:43.406921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerDied","Data":"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"}
Sep 08 13:47:44.414008 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.413971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerStarted","Data":"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"}
Sep 08 13:47:44.414462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.414030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerStarted","Data":"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"}
Sep 08 13:47:44.414462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.414288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:44.414462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.414434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:44.415806 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.415780    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:47:44.439930 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:44.439885    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podStartSLOduration=3.454245152 podStartE2EDuration="7.439872283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.985627131s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:44.438373845 +0000 UTC m=+1860.389027797" watchObservedRunningTime="2026-09-08 13:47:44.439872283 +0000 UTC m=+1860.390526235"
Sep 08 13:47:45.343592 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:45.343548    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:47:45.419522 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:45.419483    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:47:46.348251 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:46.348209    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:47:50.424119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:50.424089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:47:50.424656 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:50.424607    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:47:55.343395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:55.343359    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:47:56.348458 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:47:56.348411    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:48:00.424519 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:00.424480    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:05.342910 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:05.342870    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:48:06.347641 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:06.347603    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:48:10.425283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:10.425197    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:15.343605 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:15.343558    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:48:16.348148 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:16.348106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:48:20.425240 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:20.425203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:25.342967 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:25.342921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:48:26.348467 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:26.348426    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:48:30.425576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:30.425533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:35.342840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:35.342799    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:48:36.347548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:36.347510    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:48:40.425495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:40.425455    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:45.343326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:45.343293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:48:46.348306 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:46.348276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:48:50.425357 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:50.425319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:51.531123 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:51.531077    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:48:52.819541 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:52.819509    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:48:52.819957 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:52.819839    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" containerID="cri-o://d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c" gracePeriod=30
Sep 08 13:48:52.819957 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:52.819915    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kube-rbac-proxy" containerID="cri-o://a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b" gracePeriod=30
Sep 08 13:48:53.766734 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:53.766700    2580 generic.go:356] "Generic (PLEG): container finished" podID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerID="a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b" exitCode=2
Sep 08 13:48:53.766913 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:53.766779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerDied","Data":"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"}
Sep 08 13:48:55.338551 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:55.338503    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 08 13:48:55.342875 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:55.342843    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 08 13:48:57.572487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.572450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:48:57.614456 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location\") pod \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") "
Sep 08 13:48:57.614601 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls\") pod \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") "
Sep 08 13:48:57.614601 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert\") pod \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") "
Sep 08 13:48:57.614601 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh22p\" (UniqueName: \"kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p\") pod \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") "
Sep 08 13:48:57.614789 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98cccc1d-bf8d-4a54-aa2a-0c571581da99" (UID: "98cccc1d-bf8d-4a54-aa2a-0c571581da99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:57.614915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.614889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98cccc1d-bf8d-4a54-aa2a-0c571581da99" (UID: "98cccc1d-bf8d-4a54-aa2a-0c571581da99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:57.616668 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.616637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p" (OuterVolumeSpecName: "kube-api-access-bh22p") pod "98cccc1d-bf8d-4a54-aa2a-0c571581da99" (UID: "98cccc1d-bf8d-4a54-aa2a-0c571581da99"). InnerVolumeSpecName "kube-api-access-bh22p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:57.616765 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.616698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98cccc1d-bf8d-4a54-aa2a-0c571581da99" (UID: "98cccc1d-bf8d-4a54-aa2a-0c571581da99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:48:57.715034 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config\") pod \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\" (UID: \"98cccc1d-bf8d-4a54-aa2a-0c571581da99\") "
Sep 08 13:48:57.715246 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715232    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:48:57.715302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715248    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cccc1d-bf8d-4a54-aa2a-0c571581da99-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:48:57.715302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715258    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:48:57.715302 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715266    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh22p\" (UniqueName: \"kubernetes.io/projected/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-api-access-bh22p\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:48:57.715402 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.715332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98cccc1d-bf8d-4a54-aa2a-0c571581da99" (UID: "98cccc1d-bf8d-4a54-aa2a-0c571581da99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:57.789608 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.789573    2580 generic.go:356] "Generic (PLEG): container finished" podID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerID="d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c" exitCode=0
Sep 08 13:48:57.789759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.789641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"
Sep 08 13:48:57.789759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.789656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerDied","Data":"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"}
Sep 08 13:48:57.789759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.789720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9" event={"ID":"98cccc1d-bf8d-4a54-aa2a-0c571581da99","Type":"ContainerDied","Data":"6e9df0c5582b66b8bab862a92098d722148121b1f6a9e68c642287b9d891a7fd"}
Sep 08 13:48:57.789759 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.789747    2580 scope.go:117] "RemoveContainer" containerID="a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"
Sep 08 13:48:57.798325 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.798307    2580 scope.go:117] "RemoveContainer" containerID="d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"
Sep 08 13:48:57.805930 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.805913    2580 scope.go:117] "RemoveContainer" containerID="1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"
Sep 08 13:48:57.812925 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.812903    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:48:57.813284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.813250    2580 scope.go:117] "RemoveContainer" containerID="a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"
Sep 08 13:48:57.813572 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:48:57.813551    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b\": container with ID starting with a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b not found: ID does not exist" containerID="a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"
Sep 08 13:48:57.813657 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.813585    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b"} err="failed to get container status \"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b\": rpc error: code = NotFound desc = could not find container \"a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b\": container with ID starting with a983b7a1ae4340407c9d842bdee66db4545d14ed386e8e4877e48e8db7c9c55b not found: ID does not exist"
Sep 08 13:48:57.813657 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.813612    2580 scope.go:117] "RemoveContainer" containerID="d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"
Sep 08 13:48:57.813882 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:48:57.813866    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c\": container with ID starting with d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c not found: ID does not exist" containerID="d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"
Sep 08 13:48:57.813922 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.813889    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c"} err="failed to get container status \"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c\": rpc error: code = NotFound desc = could not find container \"d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c\": container with ID starting with d7fd80583384f8c549e1c3cc7579d443c32a5affc1afa45b5269baac68cc3d1c not found: ID does not exist"
Sep 08 13:48:57.813922 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.813903    2580 scope.go:117] "RemoveContainer" containerID="1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"
Sep 08 13:48:57.814114 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:48:57.814098    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358\": container with ID starting with 1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358 not found: ID does not exist" containerID="1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"
Sep 08 13:48:57.814154 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.814117    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358"} err="failed to get container status \"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358\": rpc error: code = NotFound desc = could not find container \"1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358\": container with ID starting with 1640692c0674d25d9ad8e2305e4c3a403fb473f5cbbeb6ec1f7697425e2ac358 not found: ID does not exist"
Sep 08 13:48:57.815770 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.815747    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cccc1d-bf8d-4a54-aa2a-0c571581da99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:48:57.817417 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:57.817394    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-j2tb9"]
Sep 08 13:48:58.536407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:48:58.536375    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" path="/var/lib/kubelet/pods/98cccc1d-bf8d-4a54-aa2a-0c571581da99/volumes"
Sep 08 13:49:01.532350 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:01.532322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:49:07.796926 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:07.796895    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:49:07.797353 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:07.797109    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" containerID="cri-o://44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5" gracePeriod=30
Sep 08 13:49:07.797353 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:07.797156    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kube-rbac-proxy" containerID="cri-o://84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f" gracePeriod=30
Sep 08 13:49:08.844998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:08.844962    2580 generic.go:356] "Generic (PLEG): container finished" podID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerID="84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f" exitCode=2
Sep 08 13:49:08.844998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:08.845004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerDied","Data":"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"}
Sep 08 13:49:10.420583 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:10.420539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 08 13:49:11.531183 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:11.531139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 08 13:49:12.534384 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.534359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:49:12.635495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location\") pod \"88ad2997-d3c4-4c77-9e11-228f9d525e64\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") "
Sep 08 13:49:12.635495 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert\") pod \"88ad2997-d3c4-4c77-9e11-228f9d525e64\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") "
Sep 08 13:49:12.635727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr669\" (UniqueName: \"kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669\") pod \"88ad2997-d3c4-4c77-9e11-228f9d525e64\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") "
Sep 08 13:49:12.635727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config\") pod \"88ad2997-d3c4-4c77-9e11-228f9d525e64\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") "
Sep 08 13:49:12.635727 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls\") pod \"88ad2997-d3c4-4c77-9e11-228f9d525e64\" (UID: \"88ad2997-d3c4-4c77-9e11-228f9d525e64\") "
Sep 08 13:49:12.635894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88ad2997-d3c4-4c77-9e11-228f9d525e64" (UID: "88ad2997-d3c4-4c77-9e11-228f9d525e64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:12.635946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88ad2997-d3c4-4c77-9e11-228f9d525e64" (UID: "88ad2997-d3c4-4c77-9e11-228f9d525e64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:12.635985 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.635929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88ad2997-d3c4-4c77-9e11-228f9d525e64" (UID: "88ad2997-d3c4-4c77-9e11-228f9d525e64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:12.637572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.637547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88ad2997-d3c4-4c77-9e11-228f9d525e64" (UID: "88ad2997-d3c4-4c77-9e11-228f9d525e64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:12.637699 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.637679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669" (OuterVolumeSpecName: "kube-api-access-kr669") pod "88ad2997-d3c4-4c77-9e11-228f9d525e64" (UID: "88ad2997-d3c4-4c77-9e11-228f9d525e64"). InnerVolumeSpecName "kube-api-access-kr669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:12.736612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.736587    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr669\" (UniqueName: \"kubernetes.io/projected/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-api-access-kr669\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:49:12.736612 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.736609    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:49:12.736781 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.736619    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ad2997-d3c4-4c77-9e11-228f9d525e64-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:49:12.736781 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.736629    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ad2997-d3c4-4c77-9e11-228f9d525e64-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:49:12.736781 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.736639    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ad2997-d3c4-4c77-9e11-228f9d525e64-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:49:12.866631 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.866595    2580 generic.go:356] "Generic (PLEG): container finished" podID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerID="44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5" exitCode=0
Sep 08 13:49:12.866778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.866652    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"
Sep 08 13:49:12.866778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.866674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerDied","Data":"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"}
Sep 08 13:49:12.866778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.866747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl" event={"ID":"88ad2997-d3c4-4c77-9e11-228f9d525e64","Type":"ContainerDied","Data":"a00997caf9e8e65202ba486a5610b334e2f535b9f4f63f9d93d0ba8f0909b400"}
Sep 08 13:49:12.866916 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.866787    2580 scope.go:117] "RemoveContainer" containerID="84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"
Sep 08 13:49:12.876097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.876077    2580 scope.go:117] "RemoveContainer" containerID="44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"
Sep 08 13:49:12.884566 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.884544    2580 scope.go:117] "RemoveContainer" containerID="c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"
Sep 08 13:49:12.891615 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.891555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:49:12.892423 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.892408    2580 scope.go:117] "RemoveContainer" containerID="84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"
Sep 08 13:49:12.892690 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:49:12.892673    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f\": container with ID starting with 84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f not found: ID does not exist" containerID="84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"
Sep 08 13:49:12.892742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.892696    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f"} err="failed to get container status \"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f\": rpc error: code = NotFound desc = could not find container \"84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f\": container with ID starting with 84921b0643d96854023a94f2eaa4675ca22b74449f59866461956289c1752e7f not found: ID does not exist"
Sep 08 13:49:12.892742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.892712    2580 scope.go:117] "RemoveContainer" containerID="44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"
Sep 08 13:49:12.892959 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:49:12.892942    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5\": container with ID starting with 44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5 not found: ID does not exist" containerID="44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"
Sep 08 13:49:12.892998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.892963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5"} err="failed to get container status \"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5\": rpc error: code = NotFound desc = could not find container \"44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5\": container with ID starting with 44ea56eb286503ac6d215555c1d824d3ed45f074dc02a65ecd3b821ba6aee7d5 not found: ID does not exist"
Sep 08 13:49:12.892998 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.892977    2580 scope.go:117] "RemoveContainer" containerID="c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"
Sep 08 13:49:12.893242 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:49:12.893222    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956\": container with ID starting with c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956 not found: ID does not exist" containerID="c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"
Sep 08 13:49:12.893290 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.893249    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956"} err="failed to get container status \"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956\": rpc error: code = NotFound desc = could not find container \"c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956\": container with ID starting with c236c7cba98f17c066fdf4fb54391373abba2825d82fd95edd6569b4b6e7f956 not found: ID does not exist"
Sep 08 13:49:12.898108 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.898087    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-687875ffcb-2g8rl"]
Sep 08 13:49:12.988676 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.988638    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:49:12.989027 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989013    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="storage-initializer"
Sep 08 13:49:12.989067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989029    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="storage-initializer"
Sep 08 13:49:12.989067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989044    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container"
Sep 08 13:49:12.989067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989049    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container"
Sep 08 13:49:12.989067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989057    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container"
Sep 08 13:49:12.989067 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989076    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="storage-initializer"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989081    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="storage-initializer"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989091    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kube-rbac-proxy"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989096    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kube-rbac-proxy"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989110    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kube-rbac-proxy"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989116    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kube-rbac-proxy"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989202    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kube-rbac-proxy"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989211    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" containerName="kserve-container"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989226    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kserve-container"
Sep 08 13:49:12.989253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.989235    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cccc1d-bf8d-4a54-aa2a-0c571581da99" containerName="kube-rbac-proxy"
Sep 08 13:49:12.993914 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.993898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:12.996687 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.996664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 08 13:49:12.996830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:12.996764    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 08 13:49:13.005229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.005205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:49:13.039194 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.039156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.039315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.039202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.039315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.039241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.039315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.039289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlm9j\" (UniqueName: \"kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.039315 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.039308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140076 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140076 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlm9j\" (UniqueName: \"kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140328 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:49:13.140290    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 08 13:49:13.140481 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:49:13.140373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls podName:7936a2de-07d1-4164-aee6-04faae70bf30 nodeName:}" failed. No retries permitted until 2026-09-08 13:49:13.640350796 +0000 UTC m=+1949.591004731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 08 13:49:13.140537 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140810 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.140847 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.140828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.149649 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.149604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlm9j\" (UniqueName: \"kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.643538 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.643492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.645868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.645847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:13.903997 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:13.903912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:14.179999 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:14.179920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:49:14.183206 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:49:14.183176    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7936a2de_07d1_4164_aee6_04faae70bf30.slice/crio-b1bf153f8f6bb3585f1f7fd003490c66dfc35a4c000e0e226d0d8f77dc2e4cc0 WatchSource:0}: Error finding container b1bf153f8f6bb3585f1f7fd003490c66dfc35a4c000e0e226d0d8f77dc2e4cc0: Status 404 returned error can't find the container with id b1bf153f8f6bb3585f1f7fd003490c66dfc35a4c000e0e226d0d8f77dc2e4cc0
Sep 08 13:49:14.536043 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:14.536000    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ad2997-d3c4-4c77-9e11-228f9d525e64" path="/var/lib/kubelet/pods/88ad2997-d3c4-4c77-9e11-228f9d525e64/volumes"
Sep 08 13:49:14.878801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:14.878709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerStarted","Data":"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"}
Sep 08 13:49:14.878801 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:14.878774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerStarted","Data":"b1bf153f8f6bb3585f1f7fd003490c66dfc35a4c000e0e226d0d8f77dc2e4cc0"}
Sep 08 13:49:17.894292 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:17.894265    2580 generic.go:356] "Generic (PLEG): container finished" podID="7936a2de-07d1-4164-aee6-04faae70bf30" containerID="7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7" exitCode=0
Sep 08 13:49:17.894720 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:17.894319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerDied","Data":"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"}
Sep 08 13:49:18.901310 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.901279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerStarted","Data":"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"}
Sep 08 13:49:18.901719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.901362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerStarted","Data":"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"}
Sep 08 13:49:18.901719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.901573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:18.901719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.901629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:18.902977 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.902950    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:49:18.924936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:18.924891    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podStartSLOduration=3.302678731 podStartE2EDuration="6.924877382s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622198651s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:18.923927425 +0000 UTC m=+1954.874581377" watchObservedRunningTime="2026-09-08 13:49:18.924877382 +0000 UTC m=+1954.875531334"
Sep 08 13:49:19.907111 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:19.907064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:49:24.912943 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:24.912915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:49:24.913388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:24.913364    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:49:34.914313 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:34.914272    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:49:44.914027 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:44.913945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:49:54.914313 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:49:54.914270    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:50:04.913391 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:04.913353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:50:14.914419 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:14.914374    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:50:23.048739 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:23.048694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:50:23.049190 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:23.048974    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" containerID="cri-o://8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958" gracePeriod=30
Sep 08 13:50:23.049190 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:23.049056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kube-rbac-proxy" containerID="cri-o://a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc" gracePeriod=30
Sep 08 13:50:23.205842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:23.205807    2580 generic.go:356] "Generic (PLEG): container finished" podID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerID="a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc" exitCode=2
Sep 08 13:50:23.206002 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:23.205877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerDied","Data":"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"}
Sep 08 13:50:24.914335 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:24.914292    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:50:26.342995 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:26.342947    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 08 13:50:26.348369 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:26.348341    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 08 13:50:27.791736 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.791711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:50:27.985705 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.985667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config\") pod \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") "
Sep 08 13:50:27.985878 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.985714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert\") pod \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") "
Sep 08 13:50:27.985878 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.985735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6dxf\" (UniqueName: \"kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf\") pod \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") "
Sep 08 13:50:27.985878 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.985773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls\") pod \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") "
Sep 08 13:50:27.985878 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.985811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location\") pod \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\" (UID: \"69b525b7-94c4-4dd8-80c0-6f3460dee60f\") "
Sep 08 13:50:27.986129 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.986097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69b525b7-94c4-4dd8-80c0-6f3460dee60f" (UID: "69b525b7-94c4-4dd8-80c0-6f3460dee60f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:27.986293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.986188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69b525b7-94c4-4dd8-80c0-6f3460dee60f" (UID: "69b525b7-94c4-4dd8-80c0-6f3460dee60f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:27.986293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.986243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69b525b7-94c4-4dd8-80c0-6f3460dee60f" (UID: "69b525b7-94c4-4dd8-80c0-6f3460dee60f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:27.987837 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.987817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69b525b7-94c4-4dd8-80c0-6f3460dee60f" (UID: "69b525b7-94c4-4dd8-80c0-6f3460dee60f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:27.987896 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:27.987845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf" (OuterVolumeSpecName: "kube-api-access-x6dxf") pod "69b525b7-94c4-4dd8-80c0-6f3460dee60f" (UID: "69b525b7-94c4-4dd8-80c0-6f3460dee60f"). InnerVolumeSpecName "kube-api-access-x6dxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:28.086888 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.086855    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.086888 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.086883    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b525b7-94c4-4dd8-80c0-6f3460dee60f-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.086888 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.086892    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dxf\" (UniqueName: \"kubernetes.io/projected/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kube-api-access-x6dxf\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.087114 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.086904    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69b525b7-94c4-4dd8-80c0-6f3460dee60f-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.087114 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.086914    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b525b7-94c4-4dd8-80c0-6f3460dee60f-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.233148 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.233116    2580 generic.go:356] "Generic (PLEG): container finished" podID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerID="8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958" exitCode=0
Sep 08 13:50:28.233332 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.233178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerDied","Data":"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"}
Sep 08 13:50:28.233332 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.233186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"
Sep 08 13:50:28.233332 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.233231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7" event={"ID":"69b525b7-94c4-4dd8-80c0-6f3460dee60f","Type":"ContainerDied","Data":"18fe1806171041d8b225c471f50614294f7f7d17ac092db7f1e7a086f8460df3"}
Sep 08 13:50:28.233332 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.233259    2580 scope.go:117] "RemoveContainer" containerID="a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"
Sep 08 13:50:28.243081 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.243063    2580 scope.go:117] "RemoveContainer" containerID="8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"
Sep 08 13:50:28.251864 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.251846    2580 scope.go:117] "RemoveContainer" containerID="2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"
Sep 08 13:50:28.260645 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.260617    2580 scope.go:117] "RemoveContainer" containerID="a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"
Sep 08 13:50:28.260919 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:28.260901    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc\": container with ID starting with a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc not found: ID does not exist" containerID="a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"
Sep 08 13:50:28.260992 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.260925    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc"} err="failed to get container status \"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc\": rpc error: code = NotFound desc = could not find container \"a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc\": container with ID starting with a5b1a7d2d61cb04a48e488e6dba633a4559dfd62d300ed3c4198b6b378cfe6cc not found: ID does not exist"
Sep 08 13:50:28.260992 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.260942    2580 scope.go:117] "RemoveContainer" containerID="8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"
Sep 08 13:50:28.261193 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:28.261150    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958\": container with ID starting with 8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958 not found: ID does not exist" containerID="8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"
Sep 08 13:50:28.261262 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.261191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958"} err="failed to get container status \"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958\": rpc error: code = NotFound desc = could not find container \"8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958\": container with ID starting with 8c6fa755219b26daf41413e689da1e4e1e900b3ba995ceabfa7497dc72d68958 not found: ID does not exist"
Sep 08 13:50:28.261262 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.261208    2580 scope.go:117] "RemoveContainer" containerID="2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"
Sep 08 13:50:28.261440 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:28.261420    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098\": container with ID starting with 2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098 not found: ID does not exist" containerID="2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"
Sep 08 13:50:28.261496 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.261442    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098"} err="failed to get container status \"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098\": rpc error: code = NotFound desc = could not find container \"2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098\": container with ID starting with 2cace6ec347ae2ebc49e14ba757e77fad447d356ff7d72fa9d0bf75060463098 not found: ID does not exist"
Sep 08 13:50:28.261548 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.261489    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:50:28.266641 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.266619    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:50:28.267021 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267003    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="storage-initializer"
Sep 08 13:50:28.267110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267023    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="storage-initializer"
Sep 08 13:50:28.267110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kube-rbac-proxy"
Sep 08 13:50:28.267110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kube-rbac-proxy"
Sep 08 13:50:28.267110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container"
Sep 08 13:50:28.267110 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267083    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container"
Sep 08 13:50:28.267497 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267194    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kube-rbac-proxy"
Sep 08 13:50:28.267497 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.267207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" containerName="kserve-container"
Sep 08 13:50:28.272118 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.272099    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-httr7"]
Sep 08 13:50:28.272251 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.272234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.274791 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.274770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:50:28.274876 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.274796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 08 13:50:28.282151 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.282134    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:50:28.384412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.384379    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:50:28.387948 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.387931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.389729 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.389709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.389822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.389736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgc9k\" (UniqueName: \"kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.389822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.389776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.389912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.389864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.389912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.389894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.391119 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.391105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:50:28.404421 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.404396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:50:28.490750 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.490912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.490912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.490912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.490912 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgc9k\" (UniqueName: \"kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.490982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc82\" (UniqueName: \"kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.491080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.491033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.491111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491289 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.491112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.491631 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.491581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.491693 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.491657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.493298 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.493280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.499944 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.499926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgc9k\" (UniqueName: \"kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-64kcm\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.539500 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.539468    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b525b7-94c4-4dd8-80c0-6f3460dee60f" path="/var/lib/kubelet/pods/69b525b7-94c4-4dd8-80c0-6f3460dee60f/volumes"
Sep 08 13:50:28.583645 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.583573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:28.592338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc82\" (UniqueName: \"kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.592405 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.592453 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.592453 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.592453 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.592805 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.592785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.593196 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.593149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.593274 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.593234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.594976 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.594956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.601713 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.601690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc82\" (UniqueName: \"kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-25p9b\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.703462 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.703424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:28.863830 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.863803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:50:28.865912 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:50:28.865888    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac82fd0_1379_4709_906c_df106fb0f636.slice/crio-0e7eb75cbfa1d5f0520cf85714f0b93d20653f1557308c41d131cd799b55e6ac WatchSource:0}: Error finding container 0e7eb75cbfa1d5f0520cf85714f0b93d20653f1557308c41d131cd799b55e6ac: Status 404 returned error can't find the container with id 0e7eb75cbfa1d5f0520cf85714f0b93d20653f1557308c41d131cd799b55e6ac
Sep 08 13:50:28.983051 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:28.982973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:50:28.985300 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:50:28.985272    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38769e9_6318_4686_b24e_e6e1ca862acb.slice/crio-90cb95c8cfa4d3c5ee6b3d517f4ef9d222561adfb90d5fd7124702c6cddf6801 WatchSource:0}: Error finding container 90cb95c8cfa4d3c5ee6b3d517f4ef9d222561adfb90d5fd7124702c6cddf6801: Status 404 returned error can't find the container with id 90cb95c8cfa4d3c5ee6b3d517f4ef9d222561adfb90d5fd7124702c6cddf6801
Sep 08 13:50:29.240463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:29.240428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerStarted","Data":"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"}
Sep 08 13:50:29.240645 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:29.240494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerStarted","Data":"0e7eb75cbfa1d5f0520cf85714f0b93d20653f1557308c41d131cd799b55e6ac"}
Sep 08 13:50:29.242321 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:29.242283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerStarted","Data":"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"}
Sep 08 13:50:29.242463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:29.242357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerStarted","Data":"90cb95c8cfa4d3c5ee6b3d517f4ef9d222561adfb90d5fd7124702c6cddf6801"}
Sep 08 13:50:33.268078 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:33.268043    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac82fd0-1379-4709-906c-df106fb0f636" containerID="2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6" exitCode=0
Sep 08 13:50:33.268563 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:33.268120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerDied","Data":"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"}
Sep 08 13:50:33.269741 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:33.269722    2580 generic.go:356] "Generic (PLEG): container finished" podID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerID="f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86" exitCode=0
Sep 08 13:50:33.269835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:33.269772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerDied","Data":"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"}
Sep 08 13:50:34.278115 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.278078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerStarted","Data":"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"}
Sep 08 13:50:34.278594 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.278145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerStarted","Data":"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"}
Sep 08 13:50:34.278594 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.278357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:34.280185 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.280149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerStarted","Data":"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"}
Sep 08 13:50:34.280293 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.280210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerStarted","Data":"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"}
Sep 08 13:50:34.280487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.280459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:34.280546 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.280510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:34.281882 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.281855    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:50:34.299326 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.299283    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podStartSLOduration=2.560054295 podStartE2EDuration="6.299270968s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739216673s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:34.298052661 +0000 UTC m=+2030.248706613" watchObservedRunningTime="2026-09-08 13:50:34.299270968 +0000 UTC m=+2030.249924921"
Sep 08 13:50:34.317050 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.317012    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podStartSLOduration=2.616887067 podStartE2EDuration="6.317000103s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700113036s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:34.316631826 +0000 UTC m=+2030.267285788" watchObservedRunningTime="2026-09-08 13:50:34.317000103 +0000 UTC m=+2030.267654056"
Sep 08 13:50:34.914742 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:34.914713    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:50:35.285301 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:35.285257    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:50:35.285694 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:35.285422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:35.286850 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:35.286822    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:50:36.297698 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:36.297656    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:50:40.289578 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:40.289548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:50:40.290125 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:40.290095    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:50:41.302044 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:41.302015    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:50:41.302524 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:41.302496    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:50:43.110303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:43.110265    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:50:43.110721 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:43.110492    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" containerID="cri-o://55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1" gracePeriod=30
Sep 08 13:50:43.110721 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:43.110548    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kube-rbac-proxy" containerID="cri-o://56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b" gracePeriod=30
Sep 08 13:50:43.342257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:43.342214    2580 generic.go:356] "Generic (PLEG): container finished" podID="7936a2de-07d1-4164-aee6-04faae70bf30" containerID="56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b" exitCode=2
Sep 08 13:50:43.342425 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:43.342289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerDied","Data":"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"}
Sep 08 13:50:44.907925 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:44.907879    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 08 13:50:44.914087 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:44.914053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 08 13:50:48.079982 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.079955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:50:48.166718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.166621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config\") pod \"7936a2de-07d1-4164-aee6-04faae70bf30\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") "
Sep 08 13:50:48.166718 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.166688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlm9j\" (UniqueName: \"kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j\") pod \"7936a2de-07d1-4164-aee6-04faae70bf30\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") "
Sep 08 13:50:48.166945 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.166732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") pod \"7936a2de-07d1-4164-aee6-04faae70bf30\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") "
Sep 08 13:50:48.166945 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.166764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert\") pod \"7936a2de-07d1-4164-aee6-04faae70bf30\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") "
Sep 08 13:50:48.167064 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.166978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location\") pod \"7936a2de-07d1-4164-aee6-04faae70bf30\" (UID: \"7936a2de-07d1-4164-aee6-04faae70bf30\") "
Sep 08 13:50:48.167120 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7936a2de-07d1-4164-aee6-04faae70bf30" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:48.167303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7936a2de-07d1-4164-aee6-04faae70bf30" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:48.167383 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7936a2de-07d1-4164-aee6-04faae70bf30" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:48.167507 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167484    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:48.167625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167525    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7936a2de-07d1-4164-aee6-04faae70bf30-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:48.167625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.167540    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7936a2de-07d1-4164-aee6-04faae70bf30-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:48.168983 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.168963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j" (OuterVolumeSpecName: "kube-api-access-hlm9j") pod "7936a2de-07d1-4164-aee6-04faae70bf30" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30"). InnerVolumeSpecName "kube-api-access-hlm9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:48.169331 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.169311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7936a2de-07d1-4164-aee6-04faae70bf30" (UID: "7936a2de-07d1-4164-aee6-04faae70bf30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:48.268409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.268369    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlm9j\" (UniqueName: \"kubernetes.io/projected/7936a2de-07d1-4164-aee6-04faae70bf30-kube-api-access-hlm9j\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:48.268409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.268403    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7936a2de-07d1-4164-aee6-04faae70bf30-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:50:48.371303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.371263    2580 generic.go:356] "Generic (PLEG): container finished" podID="7936a2de-07d1-4164-aee6-04faae70bf30" containerID="55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1" exitCode=0
Sep 08 13:50:48.371303 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.371299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerDied","Data":"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"}
Sep 08 13:50:48.371635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.371347    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"
Sep 08 13:50:48.371635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.371395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8" event={"ID":"7936a2de-07d1-4164-aee6-04faae70bf30","Type":"ContainerDied","Data":"b1bf153f8f6bb3585f1f7fd003490c66dfc35a4c000e0e226d0d8f77dc2e4cc0"}
Sep 08 13:50:48.371635 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.371438    2580 scope.go:117] "RemoveContainer" containerID="56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"
Sep 08 13:50:48.380599 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.380407    2580 scope.go:117] "RemoveContainer" containerID="55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"
Sep 08 13:50:48.388074 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.388057    2580 scope.go:117] "RemoveContainer" containerID="7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"
Sep 08 13:50:48.395852 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.395831    2580 scope.go:117] "RemoveContainer" containerID="56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"
Sep 08 13:50:48.396134 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:48.396107    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b\": container with ID starting with 56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b not found: ID does not exist" containerID="56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"
Sep 08 13:50:48.396283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.396144    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b"} err="failed to get container status \"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b\": rpc error: code = NotFound desc = could not find container \"56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b\": container with ID starting with 56ff5328c168921e2c9980bbc17d69707b0a9239e6e31f10a8dac1922a84135b not found: ID does not exist"
Sep 08 13:50:48.396283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.396190    2580 scope.go:117] "RemoveContainer" containerID="55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"
Sep 08 13:50:48.396456 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:48.396440    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1\": container with ID starting with 55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1 not found: ID does not exist" containerID="55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"
Sep 08 13:50:48.396498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.396462    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1"} err="failed to get container status \"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1\": rpc error: code = NotFound desc = could not find container \"55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1\": container with ID starting with 55b9db3674aed650b87c2061cfabc1da8af82314d706b2dd983787b00e1778e1 not found: ID does not exist"
Sep 08 13:50:48.396498 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.396476    2580 scope.go:117] "RemoveContainer" containerID="7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"
Sep 08 13:50:48.396727 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:50:48.396706    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7\": container with ID starting with 7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7 not found: ID does not exist" containerID="7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"
Sep 08 13:50:48.396833 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.396730    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7"} err="failed to get container status \"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7\": rpc error: code = NotFound desc = could not find container \"7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7\": container with ID starting with 7d602b7e3ae27ed24f396301395ae05ae632c30dafbbcdc0c4cbd6be2d108ac7 not found: ID does not exist"
Sep 08 13:50:48.397461 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.397443    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:50:48.400073 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.400052    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d98795c69-9snk8"]
Sep 08 13:50:48.536031 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:48.535999    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" path="/var/lib/kubelet/pods/7936a2de-07d1-4164-aee6-04faae70bf30/volumes"
Sep 08 13:50:50.290783 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:50.290744    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:50:51.303307 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:50:51.303269    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:00.290179 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:00.290134    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:51:01.302682 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:01.302641    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:10.290228 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:10.290122    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:51:11.303404 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:11.303362    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:20.290935 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:20.290896    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:51:21.302442 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:21.302399    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:30.290369 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:30.290329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:51:31.303009 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:31.302967    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:34.134396 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134363    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="storage-initializer"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134720    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="storage-initializer"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134728    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134734    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134749    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kube-rbac-proxy"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134755    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kube-rbac-proxy"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kube-rbac-proxy"
Sep 08 13:51:34.134842 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.134829    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7936a2de-07d1-4164-aee6-04faae70bf30" containerName="kserve-container"
Sep 08 13:51:34.138150 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.138134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.140594 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.140572    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 08 13:51:34.140735 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.140714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:51:34.151631 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.151612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:51:34.257418 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.257387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.257586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.257436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.257586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.257469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.257586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.257502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7vs\" (UniqueName: \"kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.257586 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.257546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358479 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358479 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7vs\" (UniqueName: \"kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.358899 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.358858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.359276 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.359255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.359404 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.359380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.361037 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.361016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.367212 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.367192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7vs\" (UniqueName: \"kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.448413 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.448342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:34.725387 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:34.725359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:51:34.729109 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:51:34.729075    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7802fefc_ff23_49f9_8f11_f6da4011056b.slice/crio-5922dd39ae12f2cc55f281bbd7f932690afa91845d45d961a50f992f6143ffc6 WatchSource:0}: Error finding container 5922dd39ae12f2cc55f281bbd7f932690afa91845d45d961a50f992f6143ffc6: Status 404 returned error can't find the container with id 5922dd39ae12f2cc55f281bbd7f932690afa91845d45d961a50f992f6143ffc6
Sep 08 13:51:35.613000 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:35.612959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerStarted","Data":"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"}
Sep 08 13:51:35.613455 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:35.613018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerStarted","Data":"5922dd39ae12f2cc55f281bbd7f932690afa91845d45d961a50f992f6143ffc6"}
Sep 08 13:51:39.633632 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:39.633544    2580 generic.go:356] "Generic (PLEG): container finished" podID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerID="748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420" exitCode=0
Sep 08 13:51:39.633632 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:39.633594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerDied","Data":"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"}
Sep 08 13:51:39.634814 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:39.634797    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:51:40.290540 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:40.290492    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:51:41.302552 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:41.302507    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:43.658990 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:43.658912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerStarted","Data":"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"}
Sep 08 13:51:43.658990 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:43.658972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerStarted","Data":"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"}
Sep 08 13:51:43.659388 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:43.659152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:43.686392 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:43.686344    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podStartSLOduration=1.5679911180000001 podStartE2EDuration="9.686327836s" totalImagesPullingTime="3.600314544s" totalInitContainerRuntime="4.518022174s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:43.684217122 +0000 UTC m=+2099.634871073" watchObservedRunningTime="2026-09-08 13:51:43.686327836 +0000 UTC m=+2099.636981787"
Sep 08 13:51:44.663946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:44.663911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:44.665253 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:44.665230    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 08 13:51:45.530481 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:45.530435    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:51:45.669398 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:45.669356    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 08 13:51:50.290823 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:50.290794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:51:50.674681 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:50.674594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:51:50.675271 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:50.675239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 08 13:51:55.531151 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:55.531112    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:51:58.256066 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:58.256034    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:51:58.256530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:58.256283    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" containerID="cri-o://1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1" gracePeriod=30
Sep 08 13:51:58.256530 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:58.256353    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kube-rbac-proxy" containerID="cri-o://70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7" gracePeriod=30
Sep 08 13:51:58.734905 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:58.734867    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac82fd0-1379-4709-906c-df106fb0f636" containerID="70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7" exitCode=2
Sep 08 13:51:58.735083 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:51:58.734941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerDied","Data":"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"}
Sep 08 13:52:00.285934 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:00.285893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 08 13:52:00.290966 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:00.290939    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 08 13:52:00.675552 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:00.675460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:52:03.000994 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.000968    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:52:03.118637 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.118543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls\") pod \"dac82fd0-1379-4709-906c-df106fb0f636\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") "
Sep 08 13:52:03.118637 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.118587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config\") pod \"dac82fd0-1379-4709-906c-df106fb0f636\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") "
Sep 08 13:52:03.118637 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.118614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location\") pod \"dac82fd0-1379-4709-906c-df106fb0f636\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") "
Sep 08 13:52:03.118915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.118696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert\") pod \"dac82fd0-1379-4709-906c-df106fb0f636\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") "
Sep 08 13:52:03.118915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.118734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgc9k\" (UniqueName: \"kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k\") pod \"dac82fd0-1379-4709-906c-df106fb0f636\" (UID: \"dac82fd0-1379-4709-906c-df106fb0f636\") "
Sep 08 13:52:03.119049 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.119011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dac82fd0-1379-4709-906c-df106fb0f636" (UID: "dac82fd0-1379-4709-906c-df106fb0f636"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:03.119092 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.119045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dac82fd0-1379-4709-906c-df106fb0f636" (UID: "dac82fd0-1379-4709-906c-df106fb0f636"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:03.119125 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.119086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dac82fd0-1379-4709-906c-df106fb0f636" (UID: "dac82fd0-1379-4709-906c-df106fb0f636"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:03.120701 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.120681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dac82fd0-1379-4709-906c-df106fb0f636" (UID: "dac82fd0-1379-4709-906c-df106fb0f636"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:03.120822 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.120802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k" (OuterVolumeSpecName: "kube-api-access-mgc9k") pod "dac82fd0-1379-4709-906c-df106fb0f636" (UID: "dac82fd0-1379-4709-906c-df106fb0f636"). InnerVolumeSpecName "kube-api-access-mgc9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:03.220328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.220294    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:03.220328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.220324    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgc9k\" (UniqueName: \"kubernetes.io/projected/dac82fd0-1379-4709-906c-df106fb0f636-kube-api-access-mgc9k\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:03.220328 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.220334    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac82fd0-1379-4709-906c-df106fb0f636-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:03.220549 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.220343    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac82fd0-1379-4709-906c-df106fb0f636-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:03.220549 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.220352    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac82fd0-1379-4709-906c-df106fb0f636-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:03.761116 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.761080    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac82fd0-1379-4709-906c-df106fb0f636" containerID="1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1" exitCode=0
Sep 08 13:52:03.761295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.761141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"
Sep 08 13:52:03.761295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.761156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerDied","Data":"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"}
Sep 08 13:52:03.761295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.761238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm" event={"ID":"dac82fd0-1379-4709-906c-df106fb0f636","Type":"ContainerDied","Data":"0e7eb75cbfa1d5f0520cf85714f0b93d20653f1557308c41d131cd799b55e6ac"}
Sep 08 13:52:03.761295 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.761264    2580 scope.go:117] "RemoveContainer" containerID="70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"
Sep 08 13:52:03.771159 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.771141    2580 scope.go:117] "RemoveContainer" containerID="1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"
Sep 08 13:52:03.779113 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.779097    2580 scope.go:117] "RemoveContainer" containerID="2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"
Sep 08 13:52:03.786684 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.786665    2580 scope.go:117] "RemoveContainer" containerID="70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"
Sep 08 13:52:03.786997 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:03.786979    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7\": container with ID starting with 70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7 not found: ID does not exist" containerID="70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"
Sep 08 13:52:03.787077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.787009    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7"} err="failed to get container status \"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7\": rpc error: code = NotFound desc = could not find container \"70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7\": container with ID starting with 70cd0c08526c862e63bf1ddebd7834ac598c0aa6f71dd95672c2cc382449bea7 not found: ID does not exist"
Sep 08 13:52:03.787077 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.787034    2580 scope.go:117] "RemoveContainer" containerID="1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"
Sep 08 13:52:03.787346 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:03.787325    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1\": container with ID starting with 1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1 not found: ID does not exist" containerID="1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"
Sep 08 13:52:03.787395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.787353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1"} err="failed to get container status \"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1\": rpc error: code = NotFound desc = could not find container \"1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1\": container with ID starting with 1bda0a32865f1a18b50cc53cc86d559fe4ea4db5cfa2e3e6efbf08a634f2c7d1 not found: ID does not exist"
Sep 08 13:52:03.787395 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.787371    2580 scope.go:117] "RemoveContainer" containerID="2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"
Sep 08 13:52:03.787666 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:03.787645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6\": container with ID starting with 2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6 not found: ID does not exist" containerID="2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"
Sep 08 13:52:03.787722 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.787672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6"} err="failed to get container status \"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6\": rpc error: code = NotFound desc = could not find container \"2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6\": container with ID starting with 2ace537661b2cb77c3eae26682c94ee931cdd7e32ca0e7a2555fae501ed7ddb6 not found: ID does not exist"
Sep 08 13:52:03.788502 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.788482    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:52:03.792141 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:03.792122    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-64kcm"]
Sep 08 13:52:04.536697 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:04.536665    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac82fd0-1379-4709-906c-df106fb0f636" path="/var/lib/kubelet/pods/dac82fd0-1379-4709-906c-df106fb0f636/volumes"
Sep 08 13:52:15.135391 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.135357    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:52:15.135956 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.135698    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container" containerID="cri-o://5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae" gracePeriod=30
Sep 08 13:52:15.135956 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.135724    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" containerID="cri-o://46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2" gracePeriod=30
Sep 08 13:52:15.669778 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.669738    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:15.820466 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.820430    2580 generic.go:356] "Generic (PLEG): container finished" podID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerID="46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2" exitCode=2
Sep 08 13:52:15.820631 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:15.820502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerDied","Data":"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"}
Sep 08 13:52:19.111874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:19.111789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:52:19.114800 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:19.114779    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:52:20.353782 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.353747    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:52:20.354187 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354127    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="storage-initializer"
Sep 08 13:52:20.354187 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354139    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="storage-initializer"
Sep 08 13:52:20.354187 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354160    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container"
Sep 08 13:52:20.354187 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354182    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container"
Sep 08 13:52:20.354329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kube-rbac-proxy"
Sep 08 13:52:20.354329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354197    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kube-rbac-proxy"
Sep 08 13:52:20.354329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354264    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kserve-container"
Sep 08 13:52:20.354329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.354279    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac82fd0-1379-4709-906c-df106fb0f636" containerName="kube-rbac-proxy"
Sep 08 13:52:20.357403 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.357385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.360584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.360560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 08 13:52:20.360584 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.360580    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 08 13:52:20.370245 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.370222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:52:20.439832 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.439807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.439980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.439856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.439980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.439896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.439980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.439916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.439980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.439933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjjm\" (UniqueName: \"kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.540972 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.540940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.540982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjjm\" (UniqueName: \"kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541134 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541422 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541685 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.541685 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.541678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.543444 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.543424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.550860 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.550838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjjm\" (UniqueName: \"kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm\") pod \"isvc-triton-predictor-5d7db58676-7s7dj\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.667369 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.667283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:52:20.669544 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.669516    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:20.946590 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:20.946559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:52:20.949084 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:52:20.949050    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11816a10_0e6e_4f41_a9c3_4f7e2211ffde.slice/crio-300a5321e4ee198e4a7230b03c1515726cc2bea0f5fadad3c405dd789d823ecc WatchSource:0}: Error finding container 300a5321e4ee198e4a7230b03c1515726cc2bea0f5fadad3c405dd789d823ecc: Status 404 returned error can't find the container with id 300a5321e4ee198e4a7230b03c1515726cc2bea0f5fadad3c405dd789d823ecc
Sep 08 13:52:21.856199 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:21.856143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerStarted","Data":"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"}
Sep 08 13:52:21.856570 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:21.856222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerStarted","Data":"300a5321e4ee198e4a7230b03c1515726cc2bea0f5fadad3c405dd789d823ecc"}
Sep 08 13:52:25.670581 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:25.670539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:25.670955 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:25.670662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:52:25.876578 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:25.876545    2580 generic.go:356] "Generic (PLEG): container finished" podID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerID="5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522" exitCode=0
Sep 08 13:52:25.876746 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:25.876618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerDied","Data":"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"}
Sep 08 13:52:30.670227 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:30.670159    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:35.669745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:35.669698    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:40.670283 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:40.670147    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:45.669947 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.669897    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 08 13:52:45.850329 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.850296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:52:45.993532 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.993493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls\") pod \"7802fefc-ff23-49f9-8f11-f6da4011056b\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") "
Sep 08 13:52:45.993745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.993619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert\") pod \"7802fefc-ff23-49f9-8f11-f6da4011056b\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") "
Sep 08 13:52:45.993745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.993660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln7vs\" (UniqueName: \"kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs\") pod \"7802fefc-ff23-49f9-8f11-f6da4011056b\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") "
Sep 08 13:52:45.993745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.993696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location\") pod \"7802fefc-ff23-49f9-8f11-f6da4011056b\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") "
Sep 08 13:52:45.993745 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.993721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config\") pod \"7802fefc-ff23-49f9-8f11-f6da4011056b\" (UID: \"7802fefc-ff23-49f9-8f11-f6da4011056b\") "
Sep 08 13:52:45.994280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.994228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7802fefc-ff23-49f9-8f11-f6da4011056b" (UID: "7802fefc-ff23-49f9-8f11-f6da4011056b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:45.994280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.994242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7802fefc-ff23-49f9-8f11-f6da4011056b" (UID: "7802fefc-ff23-49f9-8f11-f6da4011056b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:45.996948 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:45.996921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7802fefc-ff23-49f9-8f11-f6da4011056b" (UID: "7802fefc-ff23-49f9-8f11-f6da4011056b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:46.001284 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.001256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs" (OuterVolumeSpecName: "kube-api-access-ln7vs") pod "7802fefc-ff23-49f9-8f11-f6da4011056b" (UID: "7802fefc-ff23-49f9-8f11-f6da4011056b"). InnerVolumeSpecName "kube-api-access-ln7vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:46.007784 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.007758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7802fefc-ff23-49f9-8f11-f6da4011056b" (UID: "7802fefc-ff23-49f9-8f11-f6da4011056b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:46.032078 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.032040    2580 generic.go:356] "Generic (PLEG): container finished" podID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerID="5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae" exitCode=137
Sep 08 13:52:46.032234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.032126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"
Sep 08 13:52:46.032234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.032144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerDied","Data":"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"}
Sep 08 13:52:46.032487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.032266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj" event={"ID":"7802fefc-ff23-49f9-8f11-f6da4011056b","Type":"ContainerDied","Data":"5922dd39ae12f2cc55f281bbd7f932690afa91845d45d961a50f992f6143ffc6"}
Sep 08 13:52:46.032487 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.032313    2580 scope.go:117] "RemoveContainer" containerID="46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"
Sep 08 13:52:46.046840 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.046808    2580 scope.go:117] "RemoveContainer" containerID="5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"
Sep 08 13:52:46.058614 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.058521    2580 scope.go:117] "RemoveContainer" containerID="748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"
Sep 08 13:52:46.062920 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.062872    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:52:46.065730 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.065687    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wx2qj"]
Sep 08 13:52:46.072545 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.072522    2580 scope.go:117] "RemoveContainer" containerID="46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"
Sep 08 13:52:46.072944 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:46.072918    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2\": container with ID starting with 46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2 not found: ID does not exist" containerID="46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"
Sep 08 13:52:46.073044 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.072958    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2"} err="failed to get container status \"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2\": rpc error: code = NotFound desc = could not find container \"46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2\": container with ID starting with 46bd4eb8922ac04ccfc9e52ca08c23e5fd88b31b2ff26f8f8bdbcd5365c7f3f2 not found: ID does not exist"
Sep 08 13:52:46.073044 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.072985    2580 scope.go:117] "RemoveContainer" containerID="5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"
Sep 08 13:52:46.073346 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:46.073321    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae\": container with ID starting with 5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae not found: ID does not exist" containerID="5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"
Sep 08 13:52:46.073423 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.073355    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae"} err="failed to get container status \"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae\": rpc error: code = NotFound desc = could not find container \"5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae\": container with ID starting with 5971b13b9fd7f3b2ff693cddae4ff9c63c06c34c4bde39bda6b350b635be98ae not found: ID does not exist"
Sep 08 13:52:46.073423 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.073376    2580 scope.go:117] "RemoveContainer" containerID="748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"
Sep 08 13:52:46.073761 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:52:46.073695    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420\": container with ID starting with 748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420 not found: ID does not exist" containerID="748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"
Sep 08 13:52:46.073761 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.073723    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420"} err="failed to get container status \"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420\": rpc error: code = NotFound desc = could not find container \"748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420\": container with ID starting with 748887a7b52bf36309e5c457d290ae4d680c1af41ce5aeb433147303c7f8a420 not found: ID does not exist"
Sep 08 13:52:46.095367 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.095328    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7802fefc-ff23-49f9-8f11-f6da4011056b-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:46.095367 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.095356    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:46.095367 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.095370    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7vs\" (UniqueName: \"kubernetes.io/projected/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-api-access-ln7vs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:46.095576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.095384    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7802fefc-ff23-49f9-8f11-f6da4011056b-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:46.095576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.095397    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7802fefc-ff23-49f9-8f11-f6da4011056b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:52:46.539062 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:52:46.539021    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" path="/var/lib/kubelet/pods/7802fefc-ff23-49f9-8f11-f6da4011056b/volumes"
Sep 08 13:53:28.455486 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:28.455449    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:53:28.456142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:28.455845    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy" containerID="cri-o://ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26" gracePeriod=30
Sep 08 13:53:28.456142 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:28.455805    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" containerID="cri-o://e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495" gracePeriod=30
Sep 08 13:53:31.298422 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:31.298376    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 08 13:53:33.792881 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.792845    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:53:33.793338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793308    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="storage-initializer"
Sep 08 13:53:33.793338 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793326    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="storage-initializer"
Sep 08 13:53:33.793443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container"
Sep 08 13:53:33.793443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793352    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container"
Sep 08 13:53:33.793443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793378    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy"
Sep 08 13:53:33.793443 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793385    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy"
Sep 08 13:53:33.793607 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793480    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kube-rbac-proxy"
Sep 08 13:53:33.793607 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.793493    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7802fefc-ff23-49f9-8f11-f6da4011056b" containerName="kserve-container"
Sep 08 13:53:33.806925 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.806900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:33.810409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.810209    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 08 13:53:33.810409 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.810241    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 08 13:53:33.813258 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.813153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:53:33.948723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.948646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:33.948723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.948686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:33.948723 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.948713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:33.949050 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.948790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:33.949050 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:33.948902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bd8\" (UniqueName: \"kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.049908 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.049870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bd8\" (UniqueName: \"kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.049950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.049974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.050040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050214 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.050086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050715 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.050682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050938 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.050912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.050938 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.050915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.052731 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.052712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.071229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.071203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bd8\" (UniqueName: \"kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8\") pod \"isvc-canary-stop-predictor-7b7754b5bf-sfmvp\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:34.120944 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:34.120910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:53:35.530767 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:35.530712    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 08 13:53:36.298825 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:36.298783    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 08 13:53:39.151058 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.151034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:53:39.200616 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwc82\" (UniqueName: \"kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82\") pod \"a38769e9-6318-4686-b24e-e6e1ca862acb\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") "
Sep 08 13:53:39.200807 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert\") pod \"a38769e9-6318-4686-b24e-e6e1ca862acb\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") "
Sep 08 13:53:39.200807 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location\") pod \"a38769e9-6318-4686-b24e-e6e1ca862acb\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") "
Sep 08 13:53:39.200807 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls\") pod \"a38769e9-6318-4686-b24e-e6e1ca862acb\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") "
Sep 08 13:53:39.201009 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config\") pod \"a38769e9-6318-4686-b24e-e6e1ca862acb\" (UID: \"a38769e9-6318-4686-b24e-e6e1ca862acb\") "
Sep 08 13:53:39.201009 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.200972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a38769e9-6318-4686-b24e-e6e1ca862acb" (UID: "a38769e9-6318-4686-b24e-e6e1ca862acb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:53:39.201319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.201277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a38769e9-6318-4686-b24e-e6e1ca862acb" (UID: "a38769e9-6318-4686-b24e-e6e1ca862acb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:39.201319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.201300    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:53:39.201319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.201311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a38769e9-6318-4686-b24e-e6e1ca862acb" (UID: "a38769e9-6318-4686-b24e-e6e1ca862acb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:53:39.202980 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.202954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82" (OuterVolumeSpecName: "kube-api-access-rwc82") pod "a38769e9-6318-4686-b24e-e6e1ca862acb" (UID: "a38769e9-6318-4686-b24e-e6e1ca862acb"). InnerVolumeSpecName "kube-api-access-rwc82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:53:39.203460 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.203439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a38769e9-6318-4686-b24e-e6e1ca862acb" (UID: "a38769e9-6318-4686-b24e-e6e1ca862acb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:53:39.301667 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.301635    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a38769e9-6318-4686-b24e-e6e1ca862acb-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:53:39.301667 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.301664    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a38769e9-6318-4686-b24e-e6e1ca862acb-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:53:39.301667 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.301673    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:53:39.301915 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.301683    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc82\" (UniqueName: \"kubernetes.io/projected/a38769e9-6318-4686-b24e-e6e1ca862acb-kube-api-access-rwc82\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:53:39.350872 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.350675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:53:39.367349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367284    2580 generic.go:356] "Generic (PLEG): container finished" podID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerID="ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26" exitCode=2
Sep 08 13:53:39.367349 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367312    2580 generic.go:356] "Generic (PLEG): container finished" podID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerID="e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495" exitCode=0
Sep 08 13:53:39.367521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"
Sep 08 13:53:39.367521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerDied","Data":"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"}
Sep 08 13:53:39.367521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerDied","Data":"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"}
Sep 08 13:53:39.367521 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b" event={"ID":"a38769e9-6318-4686-b24e-e6e1ca862acb","Type":"ContainerDied","Data":"90cb95c8cfa4d3c5ee6b3d517f4ef9d222561adfb90d5fd7124702c6cddf6801"}
Sep 08 13:53:39.367741 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.367525    2580 scope.go:117] "RemoveContainer" containerID="ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"
Sep 08 13:53:39.396101 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.396075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:53:39.403962 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:39.403935    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-25p9b"]
Sep 08 13:53:39.889226 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:53:39.889194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88301595_3d7a_4c14_82bf_1d2fed773255.slice/crio-51cf26b3d714d06a30d60f716d1ae50aad8eb9adbe942bbf495042f8a14cf5c0 WatchSource:0}: Error finding container 51cf26b3d714d06a30d60f716d1ae50aad8eb9adbe942bbf495042f8a14cf5c0: Status 404 returned error can't find the container with id 51cf26b3d714d06a30d60f716d1ae50aad8eb9adbe942bbf495042f8a14cf5c0
Sep 08 13:53:40.374454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:40.374420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerStarted","Data":"51cf26b3d714d06a30d60f716d1ae50aad8eb9adbe942bbf495042f8a14cf5c0"}
Sep 08 13:53:40.537583 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:40.537553    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" path="/var/lib/kubelet/pods/a38769e9-6318-4686-b24e-e6e1ca862acb/volumes"
Sep 08 13:53:40.980352 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:40.980318    2580 scope.go:117] "RemoveContainer" containerID="e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"
Sep 08 13:53:41.073719 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:41.073692    2580 scope.go:117] "RemoveContainer" containerID="f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"
Sep 08 13:53:45.084936 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.084895    2580 scope.go:117] "RemoveContainer" containerID="ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"
Sep 08 13:53:45.085324 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:53:45.085142    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26\": container with ID starting with ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26 not found: ID does not exist" containerID="ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"
Sep 08 13:53:45.085324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085187    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"} err="failed to get container status \"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26\": rpc error: code = NotFound desc = could not find container \"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26\": container with ID starting with ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26 not found: ID does not exist"
Sep 08 13:53:45.085324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085209    2580 scope.go:117] "RemoveContainer" containerID="e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"
Sep 08 13:53:45.085490 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:53:45.085454    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495\": container with ID starting with e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495 not found: ID does not exist" containerID="e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"
Sep 08 13:53:45.085490 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"} err="failed to get container status \"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495\": rpc error: code = NotFound desc = could not find container \"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495\": container with ID starting with e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495 not found: ID does not exist"
Sep 08 13:53:45.085600 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085498    2580 scope.go:117] "RemoveContainer" containerID="f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"
Sep 08 13:53:45.085721 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:53:45.085703    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86\": container with ID starting with f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86 not found: ID does not exist" containerID="f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"
Sep 08 13:53:45.085779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085724    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"} err="failed to get container status \"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86\": rpc error: code = NotFound desc = could not find container \"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86\": container with ID starting with f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86 not found: ID does not exist"
Sep 08 13:53:45.085779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085744    2580 scope.go:117] "RemoveContainer" containerID="ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"
Sep 08 13:53:45.085974 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26"} err="failed to get container status \"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26\": rpc error: code = NotFound desc = could not find container \"ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26\": container with ID starting with ee705750860b24301e75447a35f74beda822a1d466344884f13c6f7651997d26 not found: ID does not exist"
Sep 08 13:53:45.086025 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.085977    2580 scope.go:117] "RemoveContainer" containerID="e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"
Sep 08 13:53:45.086390 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.086159    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495"} err="failed to get container status \"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495\": rpc error: code = NotFound desc = could not find container \"e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495\": container with ID starting with e883323e1e58e6bc26945599dbb8c7deee02ed8de9ab3aa7d072b9179804d495 not found: ID does not exist"
Sep 08 13:53:45.086476 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.086393    2580 scope.go:117] "RemoveContainer" containerID="f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"
Sep 08 13:53:45.086621 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:45.086600    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86"} err="failed to get container status \"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86\": rpc error: code = NotFound desc = could not find container \"f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86\": container with ID starting with f20440727e1486e1f118a3553a900b4b2e6efc0ffc5c834e223e8ea580b42d86 not found: ID does not exist"
Sep 08 13:53:50.437029 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:53:50.436942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerStarted","Data":"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"}
Sep 08 13:54:19.595720 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:19.595684    2580 generic.go:356] "Generic (PLEG): container finished" podID="88301595-3d7a-4c14-82bf-1d2fed773255" containerID="84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b" exitCode=0
Sep 08 13:54:19.596140 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:19.595759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerDied","Data":"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"}
Sep 08 13:54:20.604202 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:20.604155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerStarted","Data":"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"}
Sep 08 13:54:20.604534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:20.604254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerStarted","Data":"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"}
Sep 08 13:54:20.604534 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:20.604452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:54:20.606080 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:20.606055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerStarted","Data":"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"}
Sep 08 13:54:20.625921 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:20.625847    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podStartSLOduration=42.715006117 podStartE2EDuration="47.625831831s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.910825714s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:20.625158243 +0000 UTC m=+2256.575812206" watchObservedRunningTime="2026-09-08 13:54:20.625831831 +0000 UTC m=+2256.576485782"
Sep 08 13:54:21.614839 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:21.614800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerStarted","Data":"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"}
Sep 08 13:54:21.615378 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:21.615082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:54:21.616601 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:21.616574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:21.641921 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:21.641876    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podStartSLOduration=2.8684997 podStartE2EDuration="2m1.641862834s" totalImagesPullingTime="1m54.576633082s" totalInitContainerRuntime="4.196730052s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:21.639770829 +0000 UTC m=+2257.590424782" watchObservedRunningTime="2026-09-08 13:54:21.641862834 +0000 UTC m=+2257.592516786"
Sep 08 13:54:22.620408 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:22.620368    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:22.620880 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:22.620499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:22.620880 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:22.620556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:22.621799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:22.621774    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 08 13:54:23.625441 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:23.625400    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 08 13:54:27.625309 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:27.625277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:54:27.625827 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:27.625737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:28.629946 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:28.629920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:28.630554 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:28.630535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:32.737230 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:32.737193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:54:32.737605 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:32.737419    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container" containerID="cri-o://4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49" gracePeriod=30
Sep 08 13:54:32.737605 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:32.737440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kube-rbac-proxy" containerID="cri-o://769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e" gracePeriod=30
Sep 08 13:54:33.626144 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:33.626100    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 08 13:54:33.674979 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:33.674944    2580 generic.go:356] "Generic (PLEG): container finished" podID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerID="769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e" exitCode=2
Sep 08 13:54:33.675136 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:33.675019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerDied","Data":"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"}
Sep 08 13:54:35.297099 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.297072    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:35.401138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls\") pod \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") "
Sep 08 13:54:35.401138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert\") pod \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") "
Sep 08 13:54:35.401138 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location\") pod \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") "
Sep 08 13:54:35.401446 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdjjm\" (UniqueName: \"kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm\") pod \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") "
Sep 08 13:54:35.401446 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config\") pod \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\" (UID: \"11816a10-0e6e-4f41-a9c3-4f7e2211ffde\") "
Sep 08 13:54:35.401587 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11816a10-0e6e-4f41-a9c3-4f7e2211ffde" (UID: "11816a10-0e6e-4f41-a9c3-4f7e2211ffde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:54:35.401647 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11816a10-0e6e-4f41-a9c3-4f7e2211ffde" (UID: "11816a10-0e6e-4f41-a9c3-4f7e2211ffde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:35.401647 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.401596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11816a10-0e6e-4f41-a9c3-4f7e2211ffde" (UID: "11816a10-0e6e-4f41-a9c3-4f7e2211ffde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:54:35.403351 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.403316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm" (OuterVolumeSpecName: "kube-api-access-fdjjm") pod "11816a10-0e6e-4f41-a9c3-4f7e2211ffde" (UID: "11816a10-0e6e-4f41-a9c3-4f7e2211ffde"). InnerVolumeSpecName "kube-api-access-fdjjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:35.403463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.403394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11816a10-0e6e-4f41-a9c3-4f7e2211ffde" (UID: "11816a10-0e6e-4f41-a9c3-4f7e2211ffde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:35.502834 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.502806    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:54:35.502834 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.502832    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:54:35.502834 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.502841    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:54:35.503054 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.502850    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjjm\" (UniqueName: \"kubernetes.io/projected/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-api-access-fdjjm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:54:35.503054 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.502859    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11816a10-0e6e-4f41-a9c3-4f7e2211ffde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:54:35.686606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.686524    2580 generic.go:356] "Generic (PLEG): container finished" podID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerID="4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49" exitCode=0
Sep 08 13:54:35.686606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.686568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerDied","Data":"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"}
Sep 08 13:54:35.686606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.686582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"
Sep 08 13:54:35.686867 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.686616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj" event={"ID":"11816a10-0e6e-4f41-a9c3-4f7e2211ffde","Type":"ContainerDied","Data":"300a5321e4ee198e4a7230b03c1515726cc2bea0f5fadad3c405dd789d823ecc"}
Sep 08 13:54:35.686867 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.686642    2580 scope.go:117] "RemoveContainer" containerID="769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"
Sep 08 13:54:35.697786 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.697763    2580 scope.go:117] "RemoveContainer" containerID="4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"
Sep 08 13:54:35.705816 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.705797    2580 scope.go:117] "RemoveContainer" containerID="5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"
Sep 08 13:54:35.717857 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.717831    2580 scope.go:117] "RemoveContainer" containerID="769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"
Sep 08 13:54:35.718480 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:54:35.718447    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e\": container with ID starting with 769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e not found: ID does not exist" containerID="769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"
Sep 08 13:54:35.718572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.718493    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e"} err="failed to get container status \"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e\": rpc error: code = NotFound desc = could not find container \"769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e\": container with ID starting with 769a180c0af6825e71b44e6097a374f8b8bb95bb0e587d322102388bec061a7e not found: ID does not exist"
Sep 08 13:54:35.718572 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.718516    2580 scope.go:117] "RemoveContainer" containerID="4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"
Sep 08 13:54:35.718826 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:54:35.718807    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49\": container with ID starting with 4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49 not found: ID does not exist" containerID="4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"
Sep 08 13:54:35.718870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.718833    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49"} err="failed to get container status \"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49\": rpc error: code = NotFound desc = could not find container \"4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49\": container with ID starting with 4c620bc1feb99ebc59947cfcc9a250952266f512a4f89ccae1e115fffd052b49 not found: ID does not exist"
Sep 08 13:54:35.718870 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.718850    2580 scope.go:117] "RemoveContainer" containerID="5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"
Sep 08 13:54:35.719122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.719107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:54:35.719177 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:54:35.719111    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522\": container with ID starting with 5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522 not found: ID does not exist" containerID="5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"
Sep 08 13:54:35.719220 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.719190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522"} err="failed to get container status \"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522\": rpc error: code = NotFound desc = could not find container \"5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522\": container with ID starting with 5cb97202cbcfb5d6c2645e31cd2ef81f85ca156cb11aacdc9f9603e78731c522 not found: ID does not exist"
Sep 08 13:54:35.720792 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:35.720775    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7s7dj"]
Sep 08 13:54:36.536480 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:36.536447    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" path="/var/lib/kubelet/pods/11816a10-0e6e-4f41-a9c3-4f7e2211ffde/volumes"
Sep 08 13:54:37.626182 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.626139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:37.932696 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.932616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:54:37.932977 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.932965    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.932978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.932987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.932992    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933001    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933006    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933015    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="storage-initializer"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933020    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="storage-initializer"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933027    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933030 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933045    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="storage-initializer"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933050    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="storage-initializer"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933108    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933117    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kube-rbac-proxy"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933124    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38769e9-6318-4686-b24e-e6e1ca862acb" containerName="kserve-container"
Sep 08 13:54:37.933336 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.933130    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11816a10-0e6e-4f41-a9c3-4f7e2211ffde" containerName="kserve-container"
Sep 08 13:54:37.936229 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.936211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:37.939529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.939510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 08 13:54:37.939529 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.939521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 08 13:54:37.952841 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:37.952813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:54:38.025482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.025451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.025482 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.025482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.025690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.025501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmwmt\" (UniqueName: \"kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.025690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.025612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.025690 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.025662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126305 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmwmt\" (UniqueName: \"kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126478 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.126773 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.126739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.127159 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.127122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.127280 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.127133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.128940 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.128914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.135781 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.135756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmwmt\" (UniqueName: \"kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt\") pod \"isvc-xgboost-predictor-7566d65bf7-9ldkx\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.246407 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.246366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:54:38.552944 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.552822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:54:38.555091 ip-10-0-138-123 kubenswrapper[2580]: W0908 13:54:38.555062    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48b088_4c76_4d91_a250_2c589b1964fe.slice/crio-f10d536993d2dd8d1654e9ed75c1e0f6a6b9c3f829a1e0c5ea1d41e0fb5a788d WatchSource:0}: Error finding container f10d536993d2dd8d1654e9ed75c1e0f6a6b9c3f829a1e0c5ea1d41e0fb5a788d: Status 404 returned error can't find the container with id f10d536993d2dd8d1654e9ed75c1e0f6a6b9c3f829a1e0c5ea1d41e0fb5a788d
Sep 08 13:54:38.704470 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.704430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerStarted","Data":"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"}
Sep 08 13:54:38.704914 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:38.704489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerStarted","Data":"f10d536993d2dd8d1654e9ed75c1e0f6a6b9c3f829a1e0c5ea1d41e0fb5a788d"}
Sep 08 13:54:42.730428 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:42.730384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerDied","Data":"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"}
Sep 08 13:54:42.730428 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:42.730342    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerID="87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8" exitCode=0
Sep 08 13:54:47.626133 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:47.626093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:57.626620 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:57.626566    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:54:59.753463 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:59.753427    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:54:59.753874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:59.753661    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" containerID="cri-o://cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b" gracePeriod=30
Sep 08 13:54:59.753874 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:59.753755    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" containerID="cri-o://9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065" gracePeriod=30
Sep 08 13:54:59.754002 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:54:59.753893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:55:00.838445 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:00.838417    2580 generic.go:356] "Generic (PLEG): container finished" podID="88301595-3d7a-4c14-82bf-1d2fed773255" containerID="9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065" exitCode=2
Sep 08 13:55:00.838753 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:00.838493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerDied","Data":"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"}
Sep 08 13:55:01.846120 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.846079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerStarted","Data":"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"}
Sep 08 13:55:01.846543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.846141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerStarted","Data":"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"}
Sep 08 13:55:01.846543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.846434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:55:01.846543 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.846492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:55:01.847999 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.847967    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:01.871107 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:01.871058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podStartSLOduration=3.216212331 podStartE2EDuration="24.87104353s" totalImagesPullingTime="18.018342441s" totalInitContainerRuntime="3.636488758s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:01.869122211 +0000 UTC m=+2297.819776200" watchObservedRunningTime="2026-09-08 13:55:01.87104353 +0000 UTC m=+2297.821697481"
Sep 08 13:55:02.621440 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:02.621391    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:02.850598 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:02.850560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:07.621256 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:07.621212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:07.855818 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:07.855786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:55:07.856297 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:07.856267    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:09.754271 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:09.754228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:55:12.621511 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:12.621467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:12.621921 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:12.621607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:55:17.620606 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:17.620564    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:17.856203 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:17.856140    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:19.754551 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:19.754511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:55:22.621149 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:22.621104    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:27.621082 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:27.621042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 08 13:55:27.856995 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:27.856952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:29.754206 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:29.754146    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 08 13:55:30.419625 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.419595    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:55:30.492240 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert\") pod \"88301595-3d7a-4c14-82bf-1d2fed773255\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") "
Sep 08 13:55:30.492412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config\") pod \"88301595-3d7a-4c14-82bf-1d2fed773255\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") "
Sep 08 13:55:30.492412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls\") pod \"88301595-3d7a-4c14-82bf-1d2fed773255\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") "
Sep 08 13:55:30.492412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location\") pod \"88301595-3d7a-4c14-82bf-1d2fed773255\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") "
Sep 08 13:55:30.492412 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8bd8\" (UniqueName: \"kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8\") pod \"88301595-3d7a-4c14-82bf-1d2fed773255\" (UID: \"88301595-3d7a-4c14-82bf-1d2fed773255\") "
Sep 08 13:55:30.492675 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88301595-3d7a-4c14-82bf-1d2fed773255" (UID: "88301595-3d7a-4c14-82bf-1d2fed773255"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:30.492758 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88301595-3d7a-4c14-82bf-1d2fed773255" (UID: "88301595-3d7a-4c14-82bf-1d2fed773255"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:30.492806 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.492755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88301595-3d7a-4c14-82bf-1d2fed773255" (UID: "88301595-3d7a-4c14-82bf-1d2fed773255"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:30.494472 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.494451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88301595-3d7a-4c14-82bf-1d2fed773255" (UID: "88301595-3d7a-4c14-82bf-1d2fed773255"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:55:30.494549 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.494461    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8" (OuterVolumeSpecName: "kube-api-access-j8bd8") pod "88301595-3d7a-4c14-82bf-1d2fed773255" (UID: "88301595-3d7a-4c14-82bf-1d2fed773255"). InnerVolumeSpecName "kube-api-access-j8bd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:55:30.593811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.593726    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bd8\" (UniqueName: \"kubernetes.io/projected/88301595-3d7a-4c14-82bf-1d2fed773255-kube-api-access-j8bd8\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:55:30.593811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.593757    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:55:30.593811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.593771    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88301595-3d7a-4c14-82bf-1d2fed773255-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:55:30.593811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.593785    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88301595-3d7a-4c14-82bf-1d2fed773255-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:55:30.593811 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.593798    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88301595-3d7a-4c14-82bf-1d2fed773255-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:55:30.980904 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.980872    2580 generic.go:356] "Generic (PLEG): container finished" podID="88301595-3d7a-4c14-82bf-1d2fed773255" containerID="cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b" exitCode=137
Sep 08 13:55:30.981319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.980927    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"
Sep 08 13:55:30.981319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.980946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerDied","Data":"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"}
Sep 08 13:55:30.981319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.981022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp" event={"ID":"88301595-3d7a-4c14-82bf-1d2fed773255","Type":"ContainerDied","Data":"51cf26b3d714d06a30d60f716d1ae50aad8eb9adbe942bbf495042f8a14cf5c0"}
Sep 08 13:55:30.981319 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.981050    2580 scope.go:117] "RemoveContainer" containerID="9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"
Sep 08 13:55:30.989553 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.989533    2580 scope.go:117] "RemoveContainer" containerID="cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"
Sep 08 13:55:30.997525 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:30.997506    2580 scope.go:117] "RemoveContainer" containerID="84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"
Sep 08 13:55:31.004009 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.003988    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:55:31.005257 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.005155    2580 scope.go:117] "RemoveContainer" containerID="9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"
Sep 08 13:55:31.005538 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:55:31.005520    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065\": container with ID starting with 9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065 not found: ID does not exist" containerID="9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"
Sep 08 13:55:31.005617 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.005552    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065"} err="failed to get container status \"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065\": rpc error: code = NotFound desc = could not find container \"9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065\": container with ID starting with 9ccb42f27c78331b532e31eb498fff562a43c75c0d308c532f9e24911cad1065 not found: ID does not exist"
Sep 08 13:55:31.005617 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.005577    2580 scope.go:117] "RemoveContainer" containerID="cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"
Sep 08 13:55:31.005820 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:55:31.005802    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b\": container with ID starting with cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b not found: ID does not exist" containerID="cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"
Sep 08 13:55:31.005879 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.005827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b"} err="failed to get container status \"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b\": rpc error: code = NotFound desc = could not find container \"cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b\": container with ID starting with cc21110a87a081d2984fa14f68eb5eb9baab9bc288f1391cc4d9e19108b0f40b not found: ID does not exist"
Sep 08 13:55:31.005879 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.005848    2580 scope.go:117] "RemoveContainer" containerID="84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"
Sep 08 13:55:31.006075 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:55:31.006057    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b\": container with ID starting with 84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b not found: ID does not exist" containerID="84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"
Sep 08 13:55:31.006150 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.006081    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b"} err="failed to get container status \"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b\": rpc error: code = NotFound desc = could not find container \"84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b\": container with ID starting with 84c5cc4520d41576c87a9d92ca5682099627bc19565f7ba9cc018c0e9230602b not found: ID does not exist"
Sep 08 13:55:31.009372 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:31.009352    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-sfmvp"]
Sep 08 13:55:32.536324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:32.536292    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" path="/var/lib/kubelet/pods/88301595-3d7a-4c14-82bf-1d2fed773255/volumes"
Sep 08 13:55:37.856881 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:37.856803    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:47.856324 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:47.856278    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:55:57.856826 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:55:57.856789    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 08 13:56:07.856799 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:07.856764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:56:17.984282 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:17.984245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:56:17.984894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:17.984489    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container" containerID="cri-o://de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832" gracePeriod=30
Sep 08 13:56:17.984894 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:17.984511    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kube-rbac-proxy" containerID="cri-o://e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826" gracePeriod=30
Sep 08 13:56:18.208789 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:18.208750    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerID="e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826" exitCode=2
Sep 08 13:56:18.208965 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:18.208818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerDied","Data":"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"}
Sep 08 13:56:21.549984 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.549959    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:56:21.617661 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location\") pod \"dc48b088-4c76-4d91-a250-2c589b1964fe\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") "
Sep 08 13:56:21.617835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls\") pod \"dc48b088-4c76-4d91-a250-2c589b1964fe\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") "
Sep 08 13:56:21.617835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert\") pod \"dc48b088-4c76-4d91-a250-2c589b1964fe\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") "
Sep 08 13:56:21.617835 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617775    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmwmt\" (UniqueName: \"kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt\") pod \"dc48b088-4c76-4d91-a250-2c589b1964fe\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") "
Sep 08 13:56:21.618013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config\") pod \"dc48b088-4c76-4d91-a250-2c589b1964fe\" (UID: \"dc48b088-4c76-4d91-a250-2c589b1964fe\") "
Sep 08 13:56:21.618013 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.617923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc48b088-4c76-4d91-a250-2c589b1964fe" (UID: "dc48b088-4c76-4d91-a250-2c589b1964fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:21.618233 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.618156    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc48b088-4c76-4d91-a250-2c589b1964fe-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:56:21.618233 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.618189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc48b088-4c76-4d91-a250-2c589b1964fe" (UID: "dc48b088-4c76-4d91-a250-2c589b1964fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:21.618340 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.618244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc48b088-4c76-4d91-a250-2c589b1964fe" (UID: "dc48b088-4c76-4d91-a250-2c589b1964fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:21.619797 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.619775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt" (OuterVolumeSpecName: "kube-api-access-vmwmt") pod "dc48b088-4c76-4d91-a250-2c589b1964fe" (UID: "dc48b088-4c76-4d91-a250-2c589b1964fe"). InnerVolumeSpecName "kube-api-access-vmwmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:21.619891 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.619800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc48b088-4c76-4d91-a250-2c589b1964fe" (UID: "dc48b088-4c76-4d91-a250-2c589b1964fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:21.719234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.719188    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc48b088-4c76-4d91-a250-2c589b1964fe-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:56:21.719234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.719229    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:56:21.719234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.719242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmwmt\" (UniqueName: \"kubernetes.io/projected/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-api-access-vmwmt\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:56:21.719454 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:21.719255    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc48b088-4c76-4d91-a250-2c589b1964fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 13:56:22.228577 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.228537    2580 generic.go:356] "Generic (PLEG): container finished" podID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerID="de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832" exitCode=0
Sep 08 13:56:22.228756 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.228594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerDied","Data":"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"}
Sep 08 13:56:22.228756 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.228603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"
Sep 08 13:56:22.228756 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.228657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx" event={"ID":"dc48b088-4c76-4d91-a250-2c589b1964fe","Type":"ContainerDied","Data":"f10d536993d2dd8d1654e9ed75c1e0f6a6b9c3f829a1e0c5ea1d41e0fb5a788d"}
Sep 08 13:56:22.228756 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.228693    2580 scope.go:117] "RemoveContainer" containerID="e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"
Sep 08 13:56:22.238097 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.238067    2580 scope.go:117] "RemoveContainer" containerID="de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"
Sep 08 13:56:22.245784 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.245763    2580 scope.go:117] "RemoveContainer" containerID="87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"
Sep 08 13:56:22.253589 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.253563    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:56:22.254231 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.254212    2580 scope.go:117] "RemoveContainer" containerID="e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"
Sep 08 13:56:22.254525 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:56:22.254506    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826\": container with ID starting with e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826 not found: ID does not exist" containerID="e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"
Sep 08 13:56:22.254576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.254534    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826"} err="failed to get container status \"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826\": rpc error: code = NotFound desc = could not find container \"e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826\": container with ID starting with e877badaea667ff717ec2ebad1f2eaeb36f5415c33b221775e0f85b1ef006826 not found: ID does not exist"
Sep 08 13:56:22.254576 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.254551    2580 scope.go:117] "RemoveContainer" containerID="de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"
Sep 08 13:56:22.254797 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:56:22.254776    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832\": container with ID starting with de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832 not found: ID does not exist" containerID="de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"
Sep 08 13:56:22.254868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.254806    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832"} err="failed to get container status \"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832\": rpc error: code = NotFound desc = could not find container \"de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832\": container with ID starting with de621a3e4cbc0ce1e117e8258e584cca525b70f3e1fcc133c3c7c2266abff832 not found: ID does not exist"
Sep 08 13:56:22.254868 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.254831    2580 scope.go:117] "RemoveContainer" containerID="87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"
Sep 08 13:56:22.255107 ip-10-0-138-123 kubenswrapper[2580]: E0908 13:56:22.255091    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8\": container with ID starting with 87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8 not found: ID does not exist" containerID="87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"
Sep 08 13:56:22.255234 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.255111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8"} err="failed to get container status \"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8\": rpc error: code = NotFound desc = could not find container \"87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8\": container with ID starting with 87123e8d5e21240e6f0c71879bd61857a8799f01a6467b8cdaf46584f16330d8 not found: ID does not exist"
Sep 08 13:56:22.257590 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.257569    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-9ldkx"]
Sep 08 13:56:22.541779 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:56:22.541695    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" path="/var/lib/kubelet/pods/dc48b088-4c76-4d91-a250-2c589b1964fe/volumes"
Sep 08 13:57:19.143122 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:57:19.143087    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 13:57:19.146913 ip-10-0-138-123 kubenswrapper[2580]: I0908 13:57:19.146895    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:00:58.254398 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254365    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254699    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254710    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="storage-initializer"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254724    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="storage-initializer"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="storage-initializer"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254737    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="storage-initializer"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254747    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254785    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254791    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254797    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kube-rbac-proxy"
Sep 08 14:00:58.254828 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254802    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kube-rbac-proxy"
Sep 08 14:00:58.255230 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254853    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kube-rbac-proxy"
Sep 08 14:00:58.255230 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254860    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kube-rbac-proxy"
Sep 08 14:00:58.255230 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254869    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88301595-3d7a-4c14-82bf-1d2fed773255" containerName="kserve-container"
Sep 08 14:00:58.255230 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.254876    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc48b088-4c76-4d91-a250-2c589b1964fe" containerName="kserve-container"
Sep 08 14:00:58.258115 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.258099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.260961 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.260937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 08 14:00:58.261221 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.261200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:00:58.261323 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.261201    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:00:58.261959 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.261943    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:00:58.262037 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.262026    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:00:58.262194 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.262149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:00:58.262265 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.262149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 08 14:00:58.270125 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.270103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:00:58.321659 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.321628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhjx\" (UniqueName: \"kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.321806 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.321677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.321806 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.321693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.321806 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.321764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.321806 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.321798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422219 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422383 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422383 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhjx\" (UniqueName: \"kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422383 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422383 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422679 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422919 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.422994 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.422938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.424716 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.424698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.432033 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.432011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhjx\" (UniqueName: \"kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx\") pod \"isvc-xgboost-v2-predictor-fcc668bc-58w2z\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.568278 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.568204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:00:58.845108 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:58.845083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:00:58.848100 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:00:58.848075    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc66745_d43e_4a3c_b9b1_e7217a1fbccc.slice/crio-93d270fb00f6a6d349a9931c9c4ef9158c149e070862fb1be65d36af660aaba9 WatchSource:0}: Error finding container 93d270fb00f6a6d349a9931c9c4ef9158c149e070862fb1be65d36af660aaba9: Status 404 returned error can't find the container with id 93d270fb00f6a6d349a9931c9c4ef9158c149e070862fb1be65d36af660aaba9
Sep 08 14:00:59.500994 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:59.500952    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerStarted","Data":"41da20a63eddc5705144949421947fdb8f4895d4aaca9f340b8131697d753e8d"}
Sep 08 14:00:59.501455 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:00:59.501014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerStarted","Data":"93d270fb00f6a6d349a9931c9c4ef9158c149e070862fb1be65d36af660aaba9"}
Sep 08 14:01:03.519968 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:03.519932    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerID="41da20a63eddc5705144949421947fdb8f4895d4aaca9f340b8131697d753e8d" exitCode=0
Sep 08 14:01:03.520423 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:03.520005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerDied","Data":"41da20a63eddc5705144949421947fdb8f4895d4aaca9f340b8131697d753e8d"}
Sep 08 14:01:04.527428 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:04.527392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerStarted","Data":"82dfe881e5678f35c4bfdd211a0140d209cb99386b89d6283e327f597a8764f2"}
Sep 08 14:01:04.527909 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:04.527463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerStarted","Data":"85869703de48daacdc50955ab9037fd64b6e7ecefae1ab28677b05b46558b73c"}
Sep 08 14:01:04.527909 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:04.527692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:01:04.549958 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:04.549900    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podStartSLOduration=2.785319506 podStartE2EDuration="6.549886595s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.764567089s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:01:04.548304565 +0000 UTC m=+2660.498958516" watchObservedRunningTime="2026-09-08 14:01:04.549886595 +0000 UTC m=+2660.500540548"
Sep 08 14:01:05.532209 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:05.532159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:01:05.533429 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:05.533403    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:06.536956 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:06.536914    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:11.541798 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:11.541766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:01:11.542296 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:11.542267    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:21.542698 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:21.542660    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:31.542564 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:31.542521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:41.542234 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:41.542124    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:01:51.543026 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:01:51.542980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:02:01.543267 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:01.543224    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:02:11.543029 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:11.542999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:02:18.358663 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:18.358632    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:02:18.359153 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:18.358871    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" containerID="cri-o://85869703de48daacdc50955ab9037fd64b6e7ecefae1ab28677b05b46558b73c" gracePeriod=30
Sep 08 14:02:18.359153 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:18.358919    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kube-rbac-proxy" containerID="cri-o://82dfe881e5678f35c4bfdd211a0140d209cb99386b89d6283e327f597a8764f2" gracePeriod=30
Sep 08 14:02:18.870310 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:18.870276    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerID="82dfe881e5678f35c4bfdd211a0140d209cb99386b89d6283e327f597a8764f2" exitCode=2
Sep 08 14:02:18.870483 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:18.870347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerDied","Data":"82dfe881e5678f35c4bfdd211a0140d209cb99386b89d6283e327f597a8764f2"}
Sep 08 14:02:19.175261 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:19.175055    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:02:19.182944 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:19.180430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:02:21.537303 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:21.537260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 08 14:02:21.542620 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:21.542593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 08 14:02:21.889882 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:21.889853    2580 generic.go:356] "Generic (PLEG): container finished" podID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerID="85869703de48daacdc50955ab9037fd64b6e7ecefae1ab28677b05b46558b73c" exitCode=0
Sep 08 14:02:21.890061 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:21.889922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerDied","Data":"85869703de48daacdc50955ab9037fd64b6e7ecefae1ab28677b05b46558b73c"}
Sep 08 14:02:22.008547 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.008523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:02:22.128531 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config\") pod \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") "
Sep 08 14:02:22.128531 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert\") pod \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") "
Sep 08 14:02:22.128531 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location\") pod \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") "
Sep 08 14:02:22.128778 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkhjx\" (UniqueName: \"kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx\") pod \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") "
Sep 08 14:02:22.128778 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls\") pod \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\" (UID: \"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc\") "
Sep 08 14:02:22.128895 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" (UID: "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:22.128895 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" (UID: "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:02:22.129008 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.128921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" (UID: "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:02:22.137032 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.130747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" (UID: "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:02:22.137032 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.130982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx" (OuterVolumeSpecName: "kube-api-access-kkhjx") pod "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" (UID: "5cc66745-d43e-4a3c-b9b1-e7217a1fbccc"). InnerVolumeSpecName "kube-api-access-kkhjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:22.229554 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.229520    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:02:22.229554 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.229548    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:02:22.229554 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.229558    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhjx\" (UniqueName: \"kubernetes.io/projected/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-api-access-kkhjx\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:02:22.229775 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.229569    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:02:22.229775 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.229577    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:02:22.897196 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.897092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z" event={"ID":"5cc66745-d43e-4a3c-b9b1-e7217a1fbccc","Type":"ContainerDied","Data":"93d270fb00f6a6d349a9931c9c4ef9158c149e070862fb1be65d36af660aaba9"}
Sep 08 14:02:22.897196 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.897183    2580 scope.go:117] "RemoveContainer" containerID="82dfe881e5678f35c4bfdd211a0140d209cb99386b89d6283e327f597a8764f2"
Sep 08 14:02:22.897196 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.897116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"
Sep 08 14:02:22.905880 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.905857    2580 scope.go:117] "RemoveContainer" containerID="85869703de48daacdc50955ab9037fd64b6e7ecefae1ab28677b05b46558b73c"
Sep 08 14:02:22.913212 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.913192    2580 scope.go:117] "RemoveContainer" containerID="41da20a63eddc5705144949421947fdb8f4895d4aaca9f340b8131697d753e8d"
Sep 08 14:02:22.917418 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.917388    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:02:22.923898 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:22.923878    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-58w2z"]
Sep 08 14:02:23.509050 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509018    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:02:23.509377 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kube-rbac-proxy"
Sep 08 14:02:23.509427 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509378    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kube-rbac-proxy"
Sep 08 14:02:23.509427 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509390    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container"
Sep 08 14:02:23.509427 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509397    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container"
Sep 08 14:02:23.509427 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509408    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="storage-initializer"
Sep 08 14:02:23.509427 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509414    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="storage-initializer"
Sep 08 14:02:23.509583 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kserve-container"
Sep 08 14:02:23.509583 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.509474    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" containerName="kube-rbac-proxy"
Sep 08 14:02:23.512584 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.512567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.515272 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.515243    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:02:23.515387 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.515245    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 08 14:02:23.515387 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.515371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:02:23.515519 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.515503    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:02:23.516289 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.516274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 08 14:02:23.516375 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.516346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:02:23.522264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.522243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:02:23.640643 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.640608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.640643 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.640642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.640845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.640677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.640845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.640749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdw4\" (UniqueName: \"kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742140 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742327 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742327 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742327 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdw4\" (UniqueName: \"kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742606 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.742790 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.742772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.744536 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.744514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.750520 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.750497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdw4\" (UniqueName: \"kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4\") pod \"isvc-sklearn-s3-predictor-79bfcbdc89-swm9r\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:23.823128 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:23.823063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:24.095463 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:24.095305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:02:24.097828 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:02:24.097802    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35465267_df08_493c_bd1a_2b727b65eca6.slice/crio-b1cb60c38af8a8691dfa68cee38f01c3181ff777b1b48aa07f2a66c579643889 WatchSource:0}: Error finding container b1cb60c38af8a8691dfa68cee38f01c3181ff777b1b48aa07f2a66c579643889: Status 404 returned error can't find the container with id b1cb60c38af8a8691dfa68cee38f01c3181ff777b1b48aa07f2a66c579643889
Sep 08 14:02:24.536382 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:24.536350    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc66745-d43e-4a3c-b9b1-e7217a1fbccc" path="/var/lib/kubelet/pods/5cc66745-d43e-4a3c-b9b1-e7217a1fbccc/volumes"
Sep 08 14:02:24.911924 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:24.911835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerStarted","Data":"da7dc550a7ff439d405613dd9ba59cde28580199da1cb307fe6bab20b6eab0e6"}
Sep 08 14:02:24.911924 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:24.911899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerStarted","Data":"b1cb60c38af8a8691dfa68cee38f01c3181ff777b1b48aa07f2a66c579643889"}
Sep 08 14:02:25.918650 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:25.918608    2580 generic.go:356] "Generic (PLEG): container finished" podID="35465267-df08-493c-bd1a-2b727b65eca6" containerID="da7dc550a7ff439d405613dd9ba59cde28580199da1cb307fe6bab20b6eab0e6" exitCode=0
Sep 08 14:02:25.918650 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:25.918652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerDied","Data":"da7dc550a7ff439d405613dd9ba59cde28580199da1cb307fe6bab20b6eab0e6"}
Sep 08 14:02:26.925923 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.925888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerStarted","Data":"ea6cd63a81fadce69464cd9b68ea66aa496004da7b827134947219b6284ecf0f"}
Sep 08 14:02:26.926362 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.925947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerStarted","Data":"c4f1808e71c30f4c081023c900581ece64cc48455d6f79039558b08c409114b7"}
Sep 08 14:02:26.926362 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.926118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:26.926362 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.926195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:26.927453 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.927424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:02:26.951066 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:26.951016    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podStartSLOduration=2.850934024 podStartE2EDuration="3.95100478s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.100070756s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:26.94767259 +0000 UTC m=+2742.898326542" watchObservedRunningTime="2026-09-08 14:02:26.95100478 +0000 UTC m=+2742.901658732"
Sep 08 14:02:27.931826 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:27.931783    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:02:32.936406 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:32.936373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:02:32.936861 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:32.936836    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:02:42.937195 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:42.937138    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:02:52.937019 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:02:52.936980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:03:02.937071 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:02.937033    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:03:12.936945 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:12.936862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:03:22.937818 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:22.937782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:03:32.936834 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:32.936794    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 08 14:03:42.937864 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:42.937834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:03:43.591255 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:43.591223    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:03:43.591505 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:43.591459    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container" containerID="cri-o://c4f1808e71c30f4c081023c900581ece64cc48455d6f79039558b08c409114b7" gracePeriod=30
Sep 08 14:03:43.591738 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:43.591504    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kube-rbac-proxy" containerID="cri-o://ea6cd63a81fadce69464cd9b68ea66aa496004da7b827134947219b6284ecf0f" gracePeriod=30
Sep 08 14:03:44.295901 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:44.295870    2580 generic.go:356] "Generic (PLEG): container finished" podID="35465267-df08-493c-bd1a-2b727b65eca6" containerID="ea6cd63a81fadce69464cd9b68ea66aa496004da7b827134947219b6284ecf0f" exitCode=2
Sep 08 14:03:44.296286 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:44.295943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerDied","Data":"ea6cd63a81fadce69464cd9b68ea66aa496004da7b827134947219b6284ecf0f"}
Sep 08 14:03:47.932573 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:47.932529    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Sep 08 14:03:48.319381 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.319350    2580 generic.go:356] "Generic (PLEG): container finished" podID="35465267-df08-493c-bd1a-2b727b65eca6" containerID="c4f1808e71c30f4c081023c900581ece64cc48455d6f79039558b08c409114b7" exitCode=0
Sep 08 14:03:48.319509 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.319418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerDied","Data":"c4f1808e71c30f4c081023c900581ece64cc48455d6f79039558b08c409114b7"}
Sep 08 14:03:48.397346 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.397323    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:03:48.549234 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location\") pod \"35465267-df08-493c-bd1a-2b727b65eca6\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") "
Sep 08 14:03:48.549392 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config\") pod \"35465267-df08-493c-bd1a-2b727b65eca6\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") "
Sep 08 14:03:48.549392 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvdw4\" (UniqueName: \"kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4\") pod \"35465267-df08-493c-bd1a-2b727b65eca6\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") "
Sep 08 14:03:48.549392 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls\") pod \"35465267-df08-493c-bd1a-2b727b65eca6\" (UID: \"35465267-df08-493c-bd1a-2b727b65eca6\") "
Sep 08 14:03:48.549559 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35465267-df08-493c-bd1a-2b727b65eca6" (UID: "35465267-df08-493c-bd1a-2b727b65eca6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:48.549626 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.549596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35465267-df08-493c-bd1a-2b727b65eca6" (UID: "35465267-df08-493c-bd1a-2b727b65eca6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:03:48.551381 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.551354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4" (OuterVolumeSpecName: "kube-api-access-jvdw4") pod "35465267-df08-493c-bd1a-2b727b65eca6" (UID: "35465267-df08-493c-bd1a-2b727b65eca6"). InnerVolumeSpecName "kube-api-access-jvdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:03:48.551471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.551429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35465267-df08-493c-bd1a-2b727b65eca6" (UID: "35465267-df08-493c-bd1a-2b727b65eca6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:03:48.650022 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.649986    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35465267-df08-493c-bd1a-2b727b65eca6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:03:48.650022 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.650017    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdw4\" (UniqueName: \"kubernetes.io/projected/35465267-df08-493c-bd1a-2b727b65eca6-kube-api-access-jvdw4\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:03:48.650226 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.650030    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35465267-df08-493c-bd1a-2b727b65eca6-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:03:48.650226 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:48.650043    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35465267-df08-493c-bd1a-2b727b65eca6-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:03:49.326384 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.326355    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"
Sep 08 14:03:49.326384 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.326359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r" event={"ID":"35465267-df08-493c-bd1a-2b727b65eca6","Type":"ContainerDied","Data":"b1cb60c38af8a8691dfa68cee38f01c3181ff777b1b48aa07f2a66c579643889"}
Sep 08 14:03:49.326900 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.326431    2580 scope.go:117] "RemoveContainer" containerID="ea6cd63a81fadce69464cd9b68ea66aa496004da7b827134947219b6284ecf0f"
Sep 08 14:03:49.335419 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.335400    2580 scope.go:117] "RemoveContainer" containerID="c4f1808e71c30f4c081023c900581ece64cc48455d6f79039558b08c409114b7"
Sep 08 14:03:49.343095 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.343075    2580 scope.go:117] "RemoveContainer" containerID="da7dc550a7ff439d405613dd9ba59cde28580199da1cb307fe6bab20b6eab0e6"
Sep 08 14:03:49.350115 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.350090    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:03:49.353777 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:49.353754    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79bfcbdc89-swm9r"]
Sep 08 14:03:50.536269 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:03:50.536235    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35465267-df08-493c-bd1a-2b727b65eca6" path="/var/lib/kubelet/pods/35465267-df08-493c-bd1a-2b727b65eca6/volumes"
Sep 08 14:05:14.948751 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.948716    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949087    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="storage-initializer"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949099    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="storage-initializer"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949109    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949125    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kube-rbac-proxy"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kube-rbac-proxy"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949202    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kserve-container"
Sep 08 14:05:14.949207 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.949210    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35465267-df08-493c-bd1a-2b727b65eca6" containerName="kube-rbac-proxy"
Sep 08 14:05:14.952149 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.952134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:14.955259 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:05:14.955259 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:05:14.955259 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:05:14.955535 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955267    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 08 14:05:14.955535 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955280    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:05:14.955535 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.955237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 08 14:05:14.963211 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:14.963184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:15.028835 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.028805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.028981 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.028848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.028981 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.028867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s84f\" (UniqueName: \"kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.028981 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.028913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130106 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130301 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130301 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130301 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s84f\" (UniqueName: \"kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130601 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.130942 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.130922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.132615 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.132599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.140680 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.140651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s84f\" (UniqueName: \"kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.262835 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.262810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:15.539669 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.539646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:15.542285 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:05:15.542259    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3a9f54_5ec2_4f77_8575_608736912d98.slice/crio-fd1c7ebfdf5a87a85831c9c991741762ab697a7f80c5d3eba68039cd7588561c WatchSource:0}: Error finding container fd1c7ebfdf5a87a85831c9c991741762ab697a7f80c5d3eba68039cd7588561c: Status 404 returned error can't find the container with id fd1c7ebfdf5a87a85831c9c991741762ab697a7f80c5d3eba68039cd7588561c
Sep 08 14:05:15.728448 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.728410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerStarted","Data":"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"}
Sep 08 14:05:15.728652 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:15.728491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerStarted","Data":"fd1c7ebfdf5a87a85831c9c991741762ab697a7f80c5d3eba68039cd7588561c"}
Sep 08 14:05:20.753378 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:20.753351    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/0.log"
Sep 08 14:05:20.753681 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:20.753390    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerID="94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f" exitCode=1
Sep 08 14:05:20.753681 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:20.753436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerDied","Data":"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"}
Sep 08 14:05:21.760391 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:21.760361    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/0.log"
Sep 08 14:05:21.760784 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:21.760422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerStarted","Data":"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"}
Sep 08 14:05:24.913888 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:24.913853    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:24.914295 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:24.914073    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer" containerID="cri-o://3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff" gracePeriod=30
Sep 08 14:05:26.156728 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.156703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/1.log"
Sep 08 14:05:26.157140 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.157123    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/0.log"
Sep 08 14:05:26.159437 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.159421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:26.214450 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location\") pod \"0e3a9f54-5ec2-4f77-8575-608736912d98\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") "
Sep 08 14:05:26.214612 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls\") pod \"0e3a9f54-5ec2-4f77-8575-608736912d98\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") "
Sep 08 14:05:26.214612 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s84f\" (UniqueName: \"kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f\") pod \"0e3a9f54-5ec2-4f77-8575-608736912d98\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") "
Sep 08 14:05:26.214612 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config\") pod \"0e3a9f54-5ec2-4f77-8575-608736912d98\" (UID: \"0e3a9f54-5ec2-4f77-8575-608736912d98\") "
Sep 08 14:05:26.214785 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214693    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e3a9f54-5ec2-4f77-8575-608736912d98" (UID: "0e3a9f54-5ec2-4f77-8575-608736912d98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:26.214903 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.214884    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e3a9f54-5ec2-4f77-8575-608736912d98" (UID: "0e3a9f54-5ec2-4f77-8575-608736912d98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:26.216448 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.216418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e3a9f54-5ec2-4f77-8575-608736912d98" (UID: "0e3a9f54-5ec2-4f77-8575-608736912d98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:05:26.216616 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.216596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f" (OuterVolumeSpecName: "kube-api-access-2s84f") pod "0e3a9f54-5ec2-4f77-8575-608736912d98" (UID: "0e3a9f54-5ec2-4f77-8575-608736912d98"). InnerVolumeSpecName "kube-api-access-2s84f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:05:26.315552 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.315485    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3a9f54-5ec2-4f77-8575-608736912d98-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:05:26.315552 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.315509    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s84f\" (UniqueName: \"kubernetes.io/projected/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-api-access-2s84f\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:05:26.315552 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.315521    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3a9f54-5ec2-4f77-8575-608736912d98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:05:26.315552 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.315530    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3a9f54-5ec2-4f77-8575-608736912d98-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:05:26.785353 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/1.log"
Sep 08 14:05:26.785728 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw_0e3a9f54-5ec2-4f77-8575-608736912d98/storage-initializer/0.log"
Sep 08 14:05:26.785819 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785748    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerID="3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff" exitCode=1
Sep 08 14:05:26.785819 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerDied","Data":"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"}
Sep 08 14:05:26.785819 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"
Sep 08 14:05:26.785937 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw" event={"ID":"0e3a9f54-5ec2-4f77-8575-608736912d98","Type":"ContainerDied","Data":"fd1c7ebfdf5a87a85831c9c991741762ab697a7f80c5d3eba68039cd7588561c"}
Sep 08 14:05:26.785937 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.785871    2580 scope.go:117] "RemoveContainer" containerID="3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"
Sep 08 14:05:26.794353 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.794332    2580 scope.go:117] "RemoveContainer" containerID="94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"
Sep 08 14:05:26.801887 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.801864    2580 scope.go:117] "RemoveContainer" containerID="3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"
Sep 08 14:05:26.802174 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:05:26.802141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff\": container with ID starting with 3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff not found: ID does not exist" containerID="3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"
Sep 08 14:05:26.802260 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.802189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff"} err="failed to get container status \"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff\": rpc error: code = NotFound desc = could not find container \"3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff\": container with ID starting with 3d68ec7713d798643eca7903a3ab9d03ff8bae8973313075ee1ab0734eb9ddff not found: ID does not exist"
Sep 08 14:05:26.802260 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.802219    2580 scope.go:117] "RemoveContainer" containerID="94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"
Sep 08 14:05:26.802455 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:05:26.802438    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f\": container with ID starting with 94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f not found: ID does not exist" containerID="94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"
Sep 08 14:05:26.802496 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.802461    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f"} err="failed to get container status \"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f\": rpc error: code = NotFound desc = could not find container \"94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f\": container with ID starting with 94e718f1e88c844d1c0f1a85ce60d5577f18b2b26c81033973661c4bdd42469f not found: ID does not exist"
Sep 08 14:05:26.822495 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.822453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:26.828029 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:26.827999    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7b9578b75c-kwbpw"]
Sep 08 14:05:28.535902 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:05:28.535870    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" path="/var/lib/kubelet/pods/0e3a9f54-5ec2-4f77-8575-608736912d98/volumes"
Sep 08 14:06:57.198272 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198232    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198620    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198634    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198639    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.198773 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.198705    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3a9f54-5ec2-4f77-8575-608736912d98" containerName="storage-initializer"
Sep 08 14:06:57.201938 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.201922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.205992 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.205970    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:06:57.205992 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.205982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 08 14:06:57.206177 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.205992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:06:57.207146 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.207126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:06:57.207264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.207124    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 08 14:06:57.207264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.207206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:06:57.213804 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.213786    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:06:57.370853 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.370818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.371070 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.370873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.371070 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.370932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.371070 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.370951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwvs\" (UniqueName: \"kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472055 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472264 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwvs\" (UniqueName: \"kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472543 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.472842 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.472821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.474495 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.474473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.485105 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.485082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwvs\" (UniqueName: \"kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.512095 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.512069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:06:57.790636 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:57.790601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:06:57.793415 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:06:57.793387    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31338fd8_333f_4af0_912e_75d6732f5b47.slice/crio-844743c4374498ca24cbd24c34958cc3d440f688507bdda4f2e64348fb628b9a WatchSource:0}: Error finding container 844743c4374498ca24cbd24c34958cc3d440f688507bdda4f2e64348fb628b9a: Status 404 returned error can't find the container with id 844743c4374498ca24cbd24c34958cc3d440f688507bdda4f2e64348fb628b9a
Sep 08 14:06:58.207736 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:58.207700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerStarted","Data":"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"}
Sep 08 14:06:58.208153 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:06:58.207780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerStarted","Data":"844743c4374498ca24cbd24c34958cc3d440f688507bdda4f2e64348fb628b9a"}
Sep 08 14:07:02.227386 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:02.227359    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/0.log"
Sep 08 14:07:02.227759 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:02.227397    2580 generic.go:356] "Generic (PLEG): container finished" podID="31338fd8-333f-4af0-912e-75d6732f5b47" containerID="40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1" exitCode=1
Sep 08 14:07:02.227759 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:02.227455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerDied","Data":"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"}
Sep 08 14:07:03.234464 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:03.234437    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/0.log"
Sep 08 14:07:03.234914 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:03.234556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerStarted","Data":"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"}
Sep 08 14:07:07.162809 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.162772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:07:07.163317 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.162983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer" containerID="cri-o://41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363" gracePeriod=30
Sep 08 14:07:07.602969 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.602947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/1.log"
Sep 08 14:07:07.603319 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.603303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/0.log"
Sep 08 14:07:07.605636 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.605618    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:07:07.663019 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.662998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtwvs\" (UniqueName: \"kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs\") pod \"31338fd8-333f-4af0-912e-75d6732f5b47\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") "
Sep 08 14:07:07.663142 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location\") pod \"31338fd8-333f-4af0-912e-75d6732f5b47\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") "
Sep 08 14:07:07.663142 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config\") pod \"31338fd8-333f-4af0-912e-75d6732f5b47\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") "
Sep 08 14:07:07.663142 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls\") pod \"31338fd8-333f-4af0-912e-75d6732f5b47\" (UID: \"31338fd8-333f-4af0-912e-75d6732f5b47\") "
Sep 08 14:07:07.663307 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31338fd8-333f-4af0-912e-75d6732f5b47" (UID: "31338fd8-333f-4af0-912e-75d6732f5b47"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:07:07.663415 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663394    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31338fd8-333f-4af0-912e-75d6732f5b47-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:07:07.663481 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.663438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31338fd8-333f-4af0-912e-75d6732f5b47" (UID: "31338fd8-333f-4af0-912e-75d6732f5b47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:07:07.665075 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.665049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs" (OuterVolumeSpecName: "kube-api-access-xtwvs") pod "31338fd8-333f-4af0-912e-75d6732f5b47" (UID: "31338fd8-333f-4af0-912e-75d6732f5b47"). InnerVolumeSpecName "kube-api-access-xtwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:07:07.665218 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.665125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31338fd8-333f-4af0-912e-75d6732f5b47" (UID: "31338fd8-333f-4af0-912e-75d6732f5b47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:07:07.764525 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.764452    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwvs\" (UniqueName: \"kubernetes.io/projected/31338fd8-333f-4af0-912e-75d6732f5b47-kube-api-access-xtwvs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:07:07.764525 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.764478    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31338fd8-333f-4af0-912e-75d6732f5b47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:07:07.764525 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:07.764488    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31338fd8-333f-4af0-912e-75d6732f5b47-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:07:08.259247 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259217    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/1.log"
Sep 08 14:07:08.259656 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht_31338fd8-333f-4af0-912e-75d6732f5b47/storage-initializer/0.log"
Sep 08 14:07:08.259711 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259651    2580 generic.go:356] "Generic (PLEG): container finished" podID="31338fd8-333f-4af0-912e-75d6732f5b47" containerID="41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363" exitCode=1
Sep 08 14:07:08.259754 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"
Sep 08 14:07:08.259754 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerDied","Data":"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"}
Sep 08 14:07:08.259826 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht" event={"ID":"31338fd8-333f-4af0-912e-75d6732f5b47","Type":"ContainerDied","Data":"844743c4374498ca24cbd24c34958cc3d440f688507bdda4f2e64348fb628b9a"}
Sep 08 14:07:08.259826 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.259800    2580 scope.go:117] "RemoveContainer" containerID="41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"
Sep 08 14:07:08.269300 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.269281    2580 scope.go:117] "RemoveContainer" containerID="40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"
Sep 08 14:07:08.277256 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.277238    2580 scope.go:117] "RemoveContainer" containerID="41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"
Sep 08 14:07:08.277544 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:07:08.277525    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363\": container with ID starting with 41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363 not found: ID does not exist" containerID="41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"
Sep 08 14:07:08.277607 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.277553    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363"} err="failed to get container status \"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363\": rpc error: code = NotFound desc = could not find container \"41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363\": container with ID starting with 41f57f408385ef4a176d050c8ebd4689e026314eb755349dacaf38ef4d639363 not found: ID does not exist"
Sep 08 14:07:08.277607 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.277570    2580 scope.go:117] "RemoveContainer" containerID="40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"
Sep 08 14:07:08.277872 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:07:08.277852    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1\": container with ID starting with 40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1 not found: ID does not exist" containerID="40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"
Sep 08 14:07:08.277964 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.277878    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1"} err="failed to get container status \"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1\": rpc error: code = NotFound desc = could not find container \"40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1\": container with ID starting with 40ef185d39ed3fb8708940b49366b6760716db5d365423cb385955c05d3379a1 not found: ID does not exist"
Sep 08 14:07:08.279967 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.279946    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:07:08.280308 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280295    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.280355 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280310    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.280355 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280326    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.280355 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280331    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.280452 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.280496 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.280487    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" containerName="storage-initializer"
Sep 08 14:07:08.284785 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.284771    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.288290 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.288269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:07:08.288541 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.288523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 08 14:07:08.288680 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.288668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:07:08.289337 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.289315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:07:08.289437 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.289349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:07:08.289725 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.289694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:07:08.289870 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.289842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 08 14:07:08.299815 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.299792    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:07:08.302830 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.302809    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:07:08.314857 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.314825    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-76cf4bb685-t2nht"]
Sep 08 14:07:08.367718 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.367690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.367845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.367737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhjf\" (UniqueName: \"kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.367845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.367766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.367845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.367799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.367955 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.367841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468245 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468369 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468369 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhjf\" (UniqueName: \"kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468369 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468551 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468787 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468946 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.468997 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.468949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.470791 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.470774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.476902 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.476877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhjf\" (UniqueName: \"kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.537490 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.537413    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31338fd8-333f-4af0-912e-75d6732f5b47" path="/var/lib/kubelet/pods/31338fd8-333f-4af0-912e-75d6732f5b47/volumes"
Sep 08 14:07:08.595532 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.595495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:08.907126 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:08.907097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:07:08.909306 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:07:08.909271    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa298997_62e2_468e_ac39_85338b2ca908.slice/crio-5951c10a0215b2b449b8d459d2755e5c45924b760be620fac18ad26c177e1aca WatchSource:0}: Error finding container 5951c10a0215b2b449b8d459d2755e5c45924b760be620fac18ad26c177e1aca: Status 404 returned error can't find the container with id 5951c10a0215b2b449b8d459d2755e5c45924b760be620fac18ad26c177e1aca
Sep 08 14:07:09.266986 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:09.266947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerStarted","Data":"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"}
Sep 08 14:07:09.267378 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:09.267003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerStarted","Data":"5951c10a0215b2b449b8d459d2755e5c45924b760be620fac18ad26c177e1aca"}
Sep 08 14:07:10.272733 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:10.272649    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa298997-62e2-468e-ac39-85338b2ca908" containerID="6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5" exitCode=0
Sep 08 14:07:10.272733 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:10.272702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerDied","Data":"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"}
Sep 08 14:07:11.279869 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.279827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerStarted","Data":"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"}
Sep 08 14:07:11.280275 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.279899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerStarted","Data":"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"}
Sep 08 14:07:11.280275 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.279932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:11.280275 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.279962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:11.281439 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.281411    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:11.303292 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:11.303247    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podStartSLOduration=2.278924436 podStartE2EDuration="3.30323243s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.024307994s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:07:11.302946479 +0000 UTC m=+3027.253600432" watchObservedRunningTime="2026-09-08 14:07:11.30323243 +0000 UTC m=+3027.253886381"
Sep 08 14:07:12.284712 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:12.284676    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:17.289081 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:17.289050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:07:17.289567 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:17.289539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:19.207069 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:19.207041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:07:19.227416 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:19.212483    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:07:27.290261 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:27.290222    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:37.289772 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:37.289683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:47.289514 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:47.289469    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:07:57.290274 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:07:57.290230    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:08:07.289719 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:07.289679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:08:17.290396 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:17.290352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 08 14:08:27.290382 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:27.290350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:08:28.300870 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:28.300831    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:08:28.301340 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:28.301105    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container" containerID="cri-o://8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792" gracePeriod=30
Sep 08 14:08:28.301340 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:28.301135    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kube-rbac-proxy" containerID="cri-o://60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc" gracePeriod=30
Sep 08 14:08:28.644474 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:28.644390    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa298997-62e2-468e-ac39-85338b2ca908" containerID="60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc" exitCode=2
Sep 08 14:08:28.644474 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:28.644459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerDied","Data":"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"}
Sep 08 14:08:29.381423 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.381391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:29.384799 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.384781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.387624 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.387596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 08 14:08:29.387739 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.387599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 08 14:08:29.398302 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.398271    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:29.420765 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.420722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.420919 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.420782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.420919 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.420863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.421003 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.420927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvcj\" (UniqueName: \"kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522135 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvcj\" (UniqueName: \"kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522305 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522305 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522305 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522487 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:29.522461    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 08 14:08:29.522551 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:29.522543    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls podName:510f98b6-b433-42a1-b7f1-9020da0b50cb nodeName:}" failed. No retries permitted until 2026-09-08 14:08:30.022520126 +0000 UTC m=+3105.973174077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" (UID: "510f98b6-b433-42a1-b7f1-9020da0b50cb") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 08 14:08:29.522623 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.522916 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.522896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:29.531544 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:29.531517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvcj\" (UniqueName: \"kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:30.026537 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.026501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:30.028991 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.028960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:30.301686 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.301595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:30.579891 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.579864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:30.581837 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:08:30.581811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510f98b6_b433_42a1_b7f1_9020da0b50cb.slice/crio-f5e26c4106341a917c603f0a547a1d17645fb0cb38608b84b43a8eb7f4fbbde4 WatchSource:0}: Error finding container f5e26c4106341a917c603f0a547a1d17645fb0cb38608b84b43a8eb7f4fbbde4: Status 404 returned error can't find the container with id f5e26c4106341a917c603f0a547a1d17645fb0cb38608b84b43a8eb7f4fbbde4
Sep 08 14:08:30.657092 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.657064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerStarted","Data":"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"}
Sep 08 14:08:30.657230 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:30.657129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerStarted","Data":"f5e26c4106341a917c603f0a547a1d17645fb0cb38608b84b43a8eb7f4fbbde4"}
Sep 08 14:08:32.285426 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:32.285383    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Sep 08 14:08:33.050215 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.050191    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:08:33.153604 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location\") pod \"aa298997-62e2-468e-ac39-85338b2ca908\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") "
Sep 08 14:08:33.153604 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert\") pod \"aa298997-62e2-468e-ac39-85338b2ca908\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") "
Sep 08 14:08:33.153832 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls\") pod \"aa298997-62e2-468e-ac39-85338b2ca908\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") "
Sep 08 14:08:33.153832 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config\") pod \"aa298997-62e2-468e-ac39-85338b2ca908\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") "
Sep 08 14:08:33.153832 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwhjf\" (UniqueName: \"kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf\") pod \"aa298997-62e2-468e-ac39-85338b2ca908\" (UID: \"aa298997-62e2-468e-ac39-85338b2ca908\") "
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.153863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa298997-62e2-468e-ac39-85338b2ca908" (UID: "aa298997-62e2-468e-ac39-85338b2ca908"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.154014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa298997-62e2-468e-ac39-85338b2ca908" (UID: "aa298997-62e2-468e-ac39-85338b2ca908"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.154007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa298997-62e2-468e-ac39-85338b2ca908" (UID: "aa298997-62e2-468e-ac39-85338b2ca908"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.154124    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa298997-62e2-468e-ac39-85338b2ca908-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.154143    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:33.154192 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.154155    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa298997-62e2-468e-ac39-85338b2ca908-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:33.155726 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.155703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa298997-62e2-468e-ac39-85338b2ca908" (UID: "aa298997-62e2-468e-ac39-85338b2ca908"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:08:33.155875 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.155858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf" (OuterVolumeSpecName: "kube-api-access-gwhjf") pod "aa298997-62e2-468e-ac39-85338b2ca908" (UID: "aa298997-62e2-468e-ac39-85338b2ca908"). InnerVolumeSpecName "kube-api-access-gwhjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:33.255600 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.255570    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa298997-62e2-468e-ac39-85338b2ca908-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:33.255600 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.255597    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhjf\" (UniqueName: \"kubernetes.io/projected/aa298997-62e2-468e-ac39-85338b2ca908-kube-api-access-gwhjf\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:33.672848 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.672813    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa298997-62e2-468e-ac39-85338b2ca908" containerID="8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792" exitCode=0
Sep 08 14:08:33.672848 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.672851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerDied","Data":"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"}
Sep 08 14:08:33.673394 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.672880    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"
Sep 08 14:08:33.673394 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.672898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp" event={"ID":"aa298997-62e2-468e-ac39-85338b2ca908","Type":"ContainerDied","Data":"5951c10a0215b2b449b8d459d2755e5c45924b760be620fac18ad26c177e1aca"}
Sep 08 14:08:33.673394 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.672924    2580 scope.go:117] "RemoveContainer" containerID="60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"
Sep 08 14:08:33.681897 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.681877    2580 scope.go:117] "RemoveContainer" containerID="8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"
Sep 08 14:08:33.689804 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.689782    2580 scope.go:117] "RemoveContainer" containerID="6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"
Sep 08 14:08:33.698056 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698034    2580 scope.go:117] "RemoveContainer" containerID="60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"
Sep 08 14:08:33.698342 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:33.698322    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc\": container with ID starting with 60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc not found: ID does not exist" containerID="60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"
Sep 08 14:08:33.698409 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698351    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc"} err="failed to get container status \"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc\": rpc error: code = NotFound desc = could not find container \"60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc\": container with ID starting with 60dec7cb38e649f64c5ace5308f60b62289c8d7f8ebfeaa5520af934aa632cdc not found: ID does not exist"
Sep 08 14:08:33.698409 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698369    2580 scope.go:117] "RemoveContainer" containerID="8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"
Sep 08 14:08:33.698660 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:33.698642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792\": container with ID starting with 8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792 not found: ID does not exist" containerID="8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"
Sep 08 14:08:33.698721 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698676    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792"} err="failed to get container status \"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792\": rpc error: code = NotFound desc = could not find container \"8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792\": container with ID starting with 8e3f9607f105a64c1476001e6bb386c7cc20ba81e225b0bba8b5cc6493091792 not found: ID does not exist"
Sep 08 14:08:33.698721 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698692    2580 scope.go:117] "RemoveContainer" containerID="6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"
Sep 08 14:08:33.698826 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:08:33.698952 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:33.698936    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5\": container with ID starting with 6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5 not found: ID does not exist" containerID="6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"
Sep 08 14:08:33.698997 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.698955    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5"} err="failed to get container status \"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5\": rpc error: code = NotFound desc = could not find container \"6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5\": container with ID starting with 6bbe658cc780b7513ff793e3026135f7911a62d3f2fd411a45af1ac00c1b28b5 not found: ID does not exist"
Sep 08 14:08:33.705099 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:33.705075    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-b8cf699f8-x6kkp"]
Sep 08 14:08:34.541926 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:34.541884    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa298997-62e2-468e-ac39-85338b2ca908" path="/var/lib/kubelet/pods/aa298997-62e2-468e-ac39-85338b2ca908/volumes"
Sep 08 14:08:36.690404 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:36.690375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/0.log"
Sep 08 14:08:36.690783 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:36.690413    2580 generic.go:356] "Generic (PLEG): container finished" podID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerID="1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490" exitCode=1
Sep 08 14:08:36.690783 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:36.690491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerDied","Data":"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"}
Sep 08 14:08:37.700580 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:37.700554    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/0.log"
Sep 08 14:08:37.700953 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:37.700653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerStarted","Data":"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"}
Sep 08 14:08:39.347400 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.347320    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:39.347794 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.347555    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer" containerID="cri-o://0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc" gracePeriod=30
Sep 08 14:08:39.475554 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.475533    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/1.log"
Sep 08 14:08:39.475948 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.475933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/0.log"
Sep 08 14:08:39.478189 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.478158    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:39.507760 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.507726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config\") pod \"510f98b6-b433-42a1-b7f1-9020da0b50cb\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") "
Sep 08 14:08:39.507896 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.507804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmvcj\" (UniqueName: \"kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj\") pod \"510f98b6-b433-42a1-b7f1-9020da0b50cb\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") "
Sep 08 14:08:39.507896 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.507844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location\") pod \"510f98b6-b433-42a1-b7f1-9020da0b50cb\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") "
Sep 08 14:08:39.507896 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.507871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") pod \"510f98b6-b433-42a1-b7f1-9020da0b50cb\" (UID: \"510f98b6-b433-42a1-b7f1-9020da0b50cb\") "
Sep 08 14:08:39.508101 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.508076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "510f98b6-b433-42a1-b7f1-9020da0b50cb" (UID: "510f98b6-b433-42a1-b7f1-9020da0b50cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:39.508199 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.508107    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "510f98b6-b433-42a1-b7f1-9020da0b50cb" (UID: "510f98b6-b433-42a1-b7f1-9020da0b50cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:08:39.509931 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.509908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj" (OuterVolumeSpecName: "kube-api-access-xmvcj") pod "510f98b6-b433-42a1-b7f1-9020da0b50cb" (UID: "510f98b6-b433-42a1-b7f1-9020da0b50cb"). InnerVolumeSpecName "kube-api-access-xmvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:39.510043 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.509949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "510f98b6-b433-42a1-b7f1-9020da0b50cb" (UID: "510f98b6-b433-42a1-b7f1-9020da0b50cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:08:39.608723 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.608641    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvcj\" (UniqueName: \"kubernetes.io/projected/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-api-access-xmvcj\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:39.608723 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.608682    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510f98b6-b433-42a1-b7f1-9020da0b50cb-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:39.608723 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.608697    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/510f98b6-b433-42a1-b7f1-9020da0b50cb-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:39.608723 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.608711    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/510f98b6-b433-42a1-b7f1-9020da0b50cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:08:39.712118 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/1.log"
Sep 08 14:08:39.712510 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg_510f98b6-b433-42a1-b7f1-9020da0b50cb/storage-initializer/0.log"
Sep 08 14:08:39.712563 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712530    2580 generic.go:356] "Generic (PLEG): container finished" podID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerID="0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc" exitCode=1
Sep 08 14:08:39.712600 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerDied","Data":"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"}
Sep 08 14:08:39.712600 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"
Sep 08 14:08:39.712670 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg" event={"ID":"510f98b6-b433-42a1-b7f1-9020da0b50cb","Type":"ContainerDied","Data":"f5e26c4106341a917c603f0a547a1d17645fb0cb38608b84b43a8eb7f4fbbde4"}
Sep 08 14:08:39.712670 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.712646    2580 scope.go:117] "RemoveContainer" containerID="0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"
Sep 08 14:08:39.722069 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.722055    2580 scope.go:117] "RemoveContainer" containerID="1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"
Sep 08 14:08:39.729479 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.729462    2580 scope.go:117] "RemoveContainer" containerID="0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"
Sep 08 14:08:39.729745 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:39.729727    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc\": container with ID starting with 0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc not found: ID does not exist" containerID="0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"
Sep 08 14:08:39.729796 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.729752    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc"} err="failed to get container status \"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc\": rpc error: code = NotFound desc = could not find container \"0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc\": container with ID starting with 0a0eee99c1cd62e6d824205b3c4c5d97b9fae99367090a58f2670e97ede653dc not found: ID does not exist"
Sep 08 14:08:39.729796 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.729767    2580 scope.go:117] "RemoveContainer" containerID="1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"
Sep 08 14:08:39.729970 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:08:39.729952    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490\": container with ID starting with 1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490 not found: ID does not exist" containerID="1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"
Sep 08 14:08:39.730008 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.729976    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490"} err="failed to get container status \"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490\": rpc error: code = NotFound desc = could not find container \"1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490\": container with ID starting with 1fe465a2923c608f05f93a06c4866f4cea1cf19f1581f955e8adf3c25e35b490 not found: ID does not exist"
Sep 08 14:08:39.754112 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.754084    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:39.756145 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:39.756125    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-9f7fb7bd4-kvxvg"]
Sep 08 14:08:40.519094 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519064    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:08:40.519471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519427    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kube-rbac-proxy"
Sep 08 14:08:40.519471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519440    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kube-rbac-proxy"
Sep 08 14:08:40.519471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519456    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container"
Sep 08 14:08:40.519471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519463    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container"
Sep 08 14:08:40.519471 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519470    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519475    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519489    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519494    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519557    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519573    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kube-rbac-proxy"
Sep 08 14:08:40.519640 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa298997-62e2-468e-ac39-85338b2ca908" containerName="kserve-container"
Sep 08 14:08:40.519882 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.519675    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" containerName="storage-initializer"
Sep 08 14:08:40.529981 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.529950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.532835 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.532809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9b44e2-kube-rbac-proxy-sar-config\""
Sep 08 14:08:40.533022 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.532824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:08:40.533127 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.532913    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:08:40.533456 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.533438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:08:40.533750 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.533460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9b44e2-predictor-serving-cert\""
Sep 08 14:08:40.533958 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.533941    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:08:40.534041 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.533974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:08:40.536504 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.536476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510f98b6-b433-42a1-b7f1-9020da0b50cb" path="/var/lib/kubelet/pods/510f98b6-b433-42a1-b7f1-9020da0b50cb/volumes"
Sep 08 14:08:40.536817 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.536745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:08:40.617884 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.617851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.618010 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.617901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.618010 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.617929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgds\" (UniqueName: \"kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.618010 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.617963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.618010 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.617987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.718389 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.718356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.718570 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.718393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgds\" (UniqueName: \"kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.718570 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.718552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.718680 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.718601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.718735 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.718701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.719090 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.719061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.719342 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.719323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.719342 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.719337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.720954 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.720931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.727824 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.727800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgds\" (UniqueName: \"kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds\") pod \"isvc-primary-9b44e2-predictor-58f964847c-nkb6l\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") " pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:40.842394 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:40.842297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:41.117402 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:41.117378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:08:41.119626 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:08:41.119594    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85af6b8f_6ecc_4b3d_a882_14682cc44494.slice/crio-55654dafd0aa2fe4700507380e243597103d9847698b9e2410c28ea3525cc7a2 WatchSource:0}: Error finding container 55654dafd0aa2fe4700507380e243597103d9847698b9e2410c28ea3525cc7a2: Status 404 returned error can't find the container with id 55654dafd0aa2fe4700507380e243597103d9847698b9e2410c28ea3525cc7a2
Sep 08 14:08:41.728127 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:41.728083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerStarted","Data":"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"}
Sep 08 14:08:41.728609 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:41.728158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerStarted","Data":"55654dafd0aa2fe4700507380e243597103d9847698b9e2410c28ea3525cc7a2"}
Sep 08 14:08:45.752141 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:45.752110    2580 generic.go:356] "Generic (PLEG): container finished" podID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerID="e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff" exitCode=0
Sep 08 14:08:45.752627 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:45.752155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerDied","Data":"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"}
Sep 08 14:08:46.759113 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:46.759074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerStarted","Data":"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"}
Sep 08 14:08:46.759479 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:46.759139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerStarted","Data":"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"}
Sep 08 14:08:46.759479 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:46.759385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:46.783613 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:46.783543    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podStartSLOduration=2.876916947 podStartE2EDuration="6.783526725s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906609778s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:08:46.781708304 +0000 UTC m=+3122.732362256" watchObservedRunningTime="2026-09-08 14:08:46.783526725 +0000 UTC m=+3122.734180675"
Sep 08 14:08:47.764358 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:47.764318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:47.765755 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:47.765724    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:08:48.769986 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:48.769945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:08:53.774220 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:53.774191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:08:53.774703 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:08:53.774677    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:03.774792 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:03.774754    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:13.774829 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:13.774725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:23.775540 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:23.775492    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:33.775579 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:33.775535    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:43.774713 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:43.774671    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:09:53.775373 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:09:53.775331    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:10:02.535570 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:02.535541    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:10:25.707845 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.707808    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:25.711630 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.711614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.714408 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.714387    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9b44e2-kube-rbac-proxy-sar-config\""
Sep 08 14:10:25.714594 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.714574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9b44e2-predictor-serving-cert\""
Sep 08 14:10:25.714672 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.714619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9b44e2\""
Sep 08 14:10:25.714672 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.714619    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9b44e2-dockercfg-dh96x\""
Sep 08 14:10:25.723368 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.723345    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:25.808053 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.808024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.808206 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.808057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.808206 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.808097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.808338 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.808207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.808338 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.808247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vj2h\" (UniqueName: \"kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909463 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909647 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vj2h\" (UniqueName: \"kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909647 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909647 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909647 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.909931 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.909900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.910314 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.910297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.910371 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.910331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.912134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.912112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:25.919620 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:25.919598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vj2h\" (UniqueName: \"kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h\") pod \"isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") " pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:26.022768 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:26.022677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:26.299991 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:26.299967    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:26.302051 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:10:26.302023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc320b7b9_9482_43cd_a729_dd461f86ed35.slice/crio-be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d WatchSource:0}: Error finding container be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d: Status 404 returned error can't find the container with id be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d
Sep 08 14:10:27.240311 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:27.240271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerStarted","Data":"abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"}
Sep 08 14:10:27.240744 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:27.240330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerStarted","Data":"be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d"}
Sep 08 14:10:31.268064 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:31.268034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/0.log"
Sep 08 14:10:31.268488 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:31.268073    2580 generic.go:356] "Generic (PLEG): container finished" podID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerID="abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616" exitCode=1
Sep 08 14:10:31.268488 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:31.268130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerDied","Data":"abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"}
Sep 08 14:10:32.275337 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:32.275308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/0.log"
Sep 08 14:10:32.275715 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:32.275397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerStarted","Data":"7bd2940c197643fed3e5ccd332bb85b28ebfc81629e162956c8d59984394a823"}
Sep 08 14:10:36.295398 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.295369    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/1.log"
Sep 08 14:10:36.295831 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.295745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/0.log"
Sep 08 14:10:36.295831 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.295777    2580 generic.go:356] "Generic (PLEG): container finished" podID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerID="7bd2940c197643fed3e5ccd332bb85b28ebfc81629e162956c8d59984394a823" exitCode=1
Sep 08 14:10:36.295831 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.295816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerDied","Data":"7bd2940c197643fed3e5ccd332bb85b28ebfc81629e162956c8d59984394a823"}
Sep 08 14:10:36.295967 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.295869    2580 scope.go:117] "RemoveContainer" containerID="abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"
Sep 08 14:10:36.296256 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.296237    2580 scope.go:117] "RemoveContainer" containerID="abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"
Sep 08 14:10:36.306927 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:10:36.306906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_kserve-ci-e2e-test(c320b7b9-9482-43cd-a729-dd461f86ed35)\"" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35"
Sep 08 14:10:36.307157 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:10:36.307131    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_kserve-ci-e2e-test_c320b7b9-9482-43cd-a729-dd461f86ed35_0 in pod sandbox be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d from index: no such id: 'abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616'" containerID="abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"
Sep 08 14:10:36.307225 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:36.307181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_kserve-ci-e2e-test_c320b7b9-9482-43cd-a729-dd461f86ed35_0 in pod sandbox be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d from index: no such id: 'abf36c4e730367c13c8dee58f78420eaccd3e482e9472176359aff80bb8d5616'"
Sep 08 14:10:37.302181 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:37.302097    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/1.log"
Sep 08 14:10:41.733364 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.733332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:41.765022 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.764988    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:10:41.765321 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.765284    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" containerID="cri-o://fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4" gracePeriod=30
Sep 08 14:10:41.765460 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.765337    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kube-rbac-proxy" containerID="cri-o://edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162" gracePeriod=30
Sep 08 14:10:41.885639 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.885620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/1.log"
Sep 08 14:10:41.887694 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.887679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:41.951505 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls\") pod \"c320b7b9-9482-43cd-a729-dd461f86ed35\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") "
Sep 08 14:10:41.951689 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert\") pod \"c320b7b9-9482-43cd-a729-dd461f86ed35\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") "
Sep 08 14:10:41.951689 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vj2h\" (UniqueName: \"kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h\") pod \"c320b7b9-9482-43cd-a729-dd461f86ed35\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") "
Sep 08 14:10:41.951689 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location\") pod \"c320b7b9-9482-43cd-a729-dd461f86ed35\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") "
Sep 08 14:10:41.951865 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config\") pod \"c320b7b9-9482-43cd-a729-dd461f86ed35\" (UID: \"c320b7b9-9482-43cd-a729-dd461f86ed35\") "
Sep 08 14:10:41.951943 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c320b7b9-9482-43cd-a729-dd461f86ed35" (UID: "c320b7b9-9482-43cd-a729-dd461f86ed35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:10:41.952004 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.951962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c320b7b9-9482-43cd-a729-dd461f86ed35" (UID: "c320b7b9-9482-43cd-a729-dd461f86ed35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:10:41.952110 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.952083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c320b7b9-9482-43cd-a729-dd461f86ed35" (UID: "c320b7b9-9482-43cd-a729-dd461f86ed35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:10:41.952253 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.952192    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:41.952253 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.952214    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c320b7b9-9482-43cd-a729-dd461f86ed35-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:41.952253 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.952231    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:41.953549 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.953529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c320b7b9-9482-43cd-a729-dd461f86ed35" (UID: "c320b7b9-9482-43cd-a729-dd461f86ed35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:10:41.953743 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:41.953725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h" (OuterVolumeSpecName: "kube-api-access-4vj2h") pod "c320b7b9-9482-43cd-a729-dd461f86ed35" (UID: "c320b7b9-9482-43cd-a729-dd461f86ed35"). InnerVolumeSpecName "kube-api-access-4vj2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:10:42.053004 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.052923    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vj2h\" (UniqueName: \"kubernetes.io/projected/c320b7b9-9482-43cd-a729-dd461f86ed35-kube-api-access-4vj2h\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:42.053004 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.052953    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c320b7b9-9482-43cd-a729-dd461f86ed35-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:42.327461 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.327372    2580 generic.go:356] "Generic (PLEG): container finished" podID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerID="edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162" exitCode=2
Sep 08 14:10:42.327609 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.327450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerDied","Data":"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"}
Sep 08 14:10:42.328642 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.328627    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf_c320b7b9-9482-43cd-a729-dd461f86ed35/storage-initializer/1.log"
Sep 08 14:10:42.328758 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.328718    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"
Sep 08 14:10:42.328758 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.328731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf" event={"ID":"c320b7b9-9482-43cd-a729-dd461f86ed35","Type":"ContainerDied","Data":"be1c218360943421462c51279be6b9683c8c3482d9d160aa9cc5226161ae360d"}
Sep 08 14:10:42.328875 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.328779    2580 scope.go:117] "RemoveContainer" containerID="7bd2940c197643fed3e5ccd332bb85b28ebfc81629e162956c8d59984394a823"
Sep 08 14:10:42.371575 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.371540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:42.381519 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.381491    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9b44e2-predictor-76dbbc848b-7hmwf"]
Sep 08 14:10:42.530511 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.530462    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 08 14:10:42.535876 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:42.535846    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" path="/var/lib/kubelet/pods/c320b7b9-9482-43cd-a729-dd461f86ed35/volumes"
Sep 08 14:10:43.771035 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:43.770991    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.71:8643/healthz\": dial tcp 10.134.0.71:8643: connect: connection refused"
Sep 08 14:10:46.609669 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.609645    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:10:46.696288 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwgds\" (UniqueName: \"kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds\") pod \"85af6b8f-6ecc-4b3d-a882-14682cc44494\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") "
Sep 08 14:10:46.696441 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config\") pod \"85af6b8f-6ecc-4b3d-a882-14682cc44494\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") "
Sep 08 14:10:46.696441 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert\") pod \"85af6b8f-6ecc-4b3d-a882-14682cc44494\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") "
Sep 08 14:10:46.696441 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location\") pod \"85af6b8f-6ecc-4b3d-a882-14682cc44494\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") "
Sep 08 14:10:46.696441 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls\") pod \"85af6b8f-6ecc-4b3d-a882-14682cc44494\" (UID: \"85af6b8f-6ecc-4b3d-a882-14682cc44494\") "
Sep 08 14:10:46.696718 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85af6b8f-6ecc-4b3d-a882-14682cc44494" (UID: "85af6b8f-6ecc-4b3d-a882-14682cc44494"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:10:46.696718 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85af6b8f-6ecc-4b3d-a882-14682cc44494" (UID: "85af6b8f-6ecc-4b3d-a882-14682cc44494"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:10:46.696884 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.696715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85af6b8f-6ecc-4b3d-a882-14682cc44494" (UID: "85af6b8f-6ecc-4b3d-a882-14682cc44494"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:10:46.698371 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.698347    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds" (OuterVolumeSpecName: "kube-api-access-gwgds") pod "85af6b8f-6ecc-4b3d-a882-14682cc44494" (UID: "85af6b8f-6ecc-4b3d-a882-14682cc44494"). InnerVolumeSpecName "kube-api-access-gwgds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:10:46.698484 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.698444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85af6b8f-6ecc-4b3d-a882-14682cc44494" (UID: "85af6b8f-6ecc-4b3d-a882-14682cc44494"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:10:46.797601 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.797563    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85af6b8f-6ecc-4b3d-a882-14682cc44494-proxy-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:46.797601 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.797596    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwgds\" (UniqueName: \"kubernetes.io/projected/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-api-access-gwgds\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:46.797810 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.797610    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:46.797810 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.797623    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85af6b8f-6ecc-4b3d-a882-14682cc44494-cabundle-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:46.797810 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:46.797635    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85af6b8f-6ecc-4b3d-a882-14682cc44494-kserve-provision-location\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Sep 08 14:10:47.354012 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.353969    2580 generic.go:356] "Generic (PLEG): container finished" podID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerID="fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4" exitCode=0
Sep 08 14:10:47.354179 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.354030    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"
Sep 08 14:10:47.354179 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.354050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerDied","Data":"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"}
Sep 08 14:10:47.354179 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.354115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l" event={"ID":"85af6b8f-6ecc-4b3d-a882-14682cc44494","Type":"ContainerDied","Data":"55654dafd0aa2fe4700507380e243597103d9847698b9e2410c28ea3525cc7a2"}
Sep 08 14:10:47.354179 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.354142    2580 scope.go:117] "RemoveContainer" containerID="edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"
Sep 08 14:10:47.362835 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.362816    2580 scope.go:117] "RemoveContainer" containerID="fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"
Sep 08 14:10:47.370175 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.370147    2580 scope.go:117] "RemoveContainer" containerID="e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"
Sep 08 14:10:47.377886 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.377855    2580 scope.go:117] "RemoveContainer" containerID="edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"
Sep 08 14:10:47.378201 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:10:47.378179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162\": container with ID starting with edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162 not found: ID does not exist" containerID="edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"
Sep 08 14:10:47.378302 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.378211    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162"} err="failed to get container status \"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162\": rpc error: code = NotFound desc = could not find container \"edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162\": container with ID starting with edc6500d0acf40dcf0224eac2430859144a0319bf55fdee38051ed8846111162 not found: ID does not exist"
Sep 08 14:10:47.378302 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.378233    2580 scope.go:117] "RemoveContainer" containerID="fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"
Sep 08 14:10:47.378523 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:10:47.378507    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4\": container with ID starting with fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4 not found: ID does not exist" containerID="fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"
Sep 08 14:10:47.378589 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.378532    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4"} err="failed to get container status \"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4\": rpc error: code = NotFound desc = could not find container \"fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4\": container with ID starting with fbd173dbf46a7c973e4b892e58e4e6acca8fe9a7e773f89b77cf48d3cd42d3a4 not found: ID does not exist"
Sep 08 14:10:47.378589 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.378552    2580 scope.go:117] "RemoveContainer" containerID="e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"
Sep 08 14:10:47.378828 ip-10-0-138-123 kubenswrapper[2580]: E0908 14:10:47.378812    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff\": container with ID starting with e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff not found: ID does not exist" containerID="e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"
Sep 08 14:10:47.378883 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.378835    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff"} err="failed to get container status \"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff\": rpc error: code = NotFound desc = could not find container \"e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff\": container with ID starting with e8cca2c6a5887f3cf6331183bc952cc478b85c41977f0782b2f32898fdf296ff not found: ID does not exist"
Sep 08 14:10:47.380511 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.380492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:10:47.384994 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:47.384973    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9b44e2-predictor-58f964847c-nkb6l"]
Sep 08 14:10:48.535962 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:10:48.535927    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" path="/var/lib/kubelet/pods/85af6b8f-6ecc-4b3d-a882-14682cc44494/volumes"
Sep 08 14:11:47.562041 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562004    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcsfc/must-gather-dxv2g"]
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562352    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562360    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kube-rbac-proxy"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562366    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kube-rbac-proxy"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562372    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562378    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562404    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="storage-initializer"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562410    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="storage-initializer"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562459    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kube-rbac-proxy"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562470    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85af6b8f-6ecc-4b3d-a882-14682cc44494" containerName="kserve-container"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562476    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.562521 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562483    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.562893 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562553    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.562893 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.562559    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c320b7b9-9482-43cd-a729-dd461f86ed35" containerName="storage-initializer"
Sep 08 14:11:47.566847 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.566822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.569588 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.569571    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcsfc\"/\"default-dockercfg-dms7m\""
Sep 08 14:11:47.569851 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.569835    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcsfc\"/\"openshift-service-ca.crt\""
Sep 08 14:11:47.570779 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.570762    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcsfc\"/\"kube-root-ca.crt\""
Sep 08 14:11:47.576088 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.576062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/must-gather-dxv2g"]
Sep 08 14:11:47.609374 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.609351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrj7\" (UniqueName: \"kubernetes.io/projected/4b896b12-98bb-4dbb-9eec-4927a34c839e-kube-api-access-mlrj7\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.609469 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.609390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b896b12-98bb-4dbb-9eec-4927a34c839e-must-gather-output\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.710048 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.710001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrj7\" (UniqueName: \"kubernetes.io/projected/4b896b12-98bb-4dbb-9eec-4927a34c839e-kube-api-access-mlrj7\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.710236 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.710069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b896b12-98bb-4dbb-9eec-4927a34c839e-must-gather-output\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.710391 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.710372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b896b12-98bb-4dbb-9eec-4927a34c839e-must-gather-output\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.719347 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.719327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrj7\" (UniqueName: \"kubernetes.io/projected/4b896b12-98bb-4dbb-9eec-4927a34c839e-kube-api-access-mlrj7\") pod \"must-gather-dxv2g\" (UID: \"4b896b12-98bb-4dbb-9eec-4927a34c839e\") " pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:47.885497 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:47.885427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/must-gather-dxv2g"
Sep 08 14:11:48.156884 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:48.156854    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/must-gather-dxv2g"]
Sep 08 14:11:48.158685 ip-10-0-138-123 kubenswrapper[2580]: W0908 14:11:48.158656    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b896b12_98bb_4dbb_9eec_4927a34c839e.slice/crio-01b49cb0cebfdd3eb5351b2d5fa5862a98b970c6e92da0e7cf691577940a6062 WatchSource:0}: Error finding container 01b49cb0cebfdd3eb5351b2d5fa5862a98b970c6e92da0e7cf691577940a6062: Status 404 returned error can't find the container with id 01b49cb0cebfdd3eb5351b2d5fa5862a98b970c6e92da0e7cf691577940a6062
Sep 08 14:11:48.160616 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:48.160596    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:11:48.635954 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:48.635912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/must-gather-dxv2g" event={"ID":"4b896b12-98bb-4dbb-9eec-4927a34c839e","Type":"ContainerStarted","Data":"01b49cb0cebfdd3eb5351b2d5fa5862a98b970c6e92da0e7cf691577940a6062"}
Sep 08 14:11:49.643992 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:49.643905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/must-gather-dxv2g" event={"ID":"4b896b12-98bb-4dbb-9eec-4927a34c839e","Type":"ContainerStarted","Data":"5379d941049c665bf5ce0f5bbdf92b0fe73c202bb9805904e3bb0fe7a3e180d6"}
Sep 08 14:11:49.643992 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:49.643964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/must-gather-dxv2g" event={"ID":"4b896b12-98bb-4dbb-9eec-4927a34c839e","Type":"ContainerStarted","Data":"1300623bd7f46c0f4a90f132bb2bd44068ed042d3e94fd78e6dfba69182ab2a9"}
Sep 08 14:11:49.662735 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:49.662665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcsfc/must-gather-dxv2g" podStartSLOduration=1.831030589 podStartE2EDuration="2.662645502s" totalImagesPullingTime="831.614913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:11:49.660063128 +0000 UTC m=+3305.610717092" watchObservedRunningTime="2026-09-08 14:11:49.662645502 +0000 UTC m=+3305.613299455"
Sep 08 14:11:50.485309 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:50.485280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fv5ww_63cd3528-9abe-4b26-8744-c662668b3f60/global-pull-secret-syncer/0.log"
Sep 08 14:11:50.723117 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:50.723067    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mkmcw_f3af11ec-f900-46e3-a79d-c54f27ef55f4/konnectivity-agent/0.log"
Sep 08 14:11:50.835739 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:50.835634    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-123.ec2.internal_b7691c42ef3bfce427afbddd33d4f4b3/haproxy/0.log"
Sep 08 14:11:55.085210 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.085156    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65789ddc66-f9zf2_bfa675da-98e3-40bc-829a-f4c166fd1b62/metrics-server/0.log"
Sep 08 14:11:55.116533 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.116495    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-zrcm7_d8f7fe01-c204-4e40-8afd-deb24e8916c0/monitoring-plugin/0.log"
Sep 08 14:11:55.319420 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.319389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq24n_db52b67a-35a7-457c-8b67-d01d9758846a/node-exporter/0.log"
Sep 08 14:11:55.342399 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.342324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq24n_db52b67a-35a7-457c-8b67-d01d9758846a/kube-rbac-proxy/0.log"
Sep 08 14:11:55.371829 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.371148    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rq24n_db52b67a-35a7-457c-8b67-d01d9758846a/init-textfile/0.log"
Sep 08 14:11:55.408293 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.408263    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wbxhc_0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7/kube-rbac-proxy-main/0.log"
Sep 08 14:11:55.433185 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.433142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wbxhc_0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7/kube-rbac-proxy-self/0.log"
Sep 08 14:11:55.459394 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.459358    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wbxhc_0574ff62-e5c2-46e2-b6a0-a2bc2f9941b7/openshift-state-metrics/0.log"
Sep 08 14:11:55.733635 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.733606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-9sbzj_3ec78bb5-b335-4874-9a24-37b4af6c260a/prometheus-operator-admission-webhook/0.log"
Sep 08 14:11:55.854216 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.854184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/thanos-query/0.log"
Sep 08 14:11:55.880842 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.880803    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/kube-rbac-proxy-web/0.log"
Sep 08 14:11:55.907818 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.907774    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/kube-rbac-proxy/0.log"
Sep 08 14:11:55.934948 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.934915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/prom-label-proxy/0.log"
Sep 08 14:11:55.963001 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.962940    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/kube-rbac-proxy-rules/0.log"
Sep 08 14:11:55.986643 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:55.986518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c574fdc9-45xxk_11fd803f-04a2-46e1-bc00-5ae70a190581/kube-rbac-proxy-metrics/0.log"
Sep 08 14:11:57.362626 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:57.362592    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qhdh8_468d9440-90f8-4110-a67b-d6aec9a871ad/networking-console-plugin/0.log"
Sep 08 14:11:58.326365 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:58.326338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bdfcf65f8-th6b5_79e813c4-fc9c-4532-b9f8-0f9a7f8c4376/console/0.log"
Sep 08 14:11:58.957105 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:58.957074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-zt762_ae6234bd-edff-4975-9758-f5780c517a8b/volume-data-source-validator/0.log"
Sep 08 14:11:59.793832 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.793795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"]
Sep 08 14:11:59.798371 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.798339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.807210 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.807125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"]
Sep 08 14:11:59.824079 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.823786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-sys\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.824079 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.823850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-lib-modules\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.824079 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.823905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-podres\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.824079 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.823933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7mb\" (UniqueName: \"kubernetes.io/projected/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-kube-api-access-dt7mb\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.824079 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.823973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-proc\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.924950 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.924916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-sys\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.924962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-lib-modules\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.924993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-podres\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7mb\" (UniqueName: \"kubernetes.io/projected/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-kube-api-access-dt7mb\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-proc\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-sys\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-lib-modules\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925134 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-podres\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.925407 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.925153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-proc\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.934017 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.933989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7mb\" (UniqueName: \"kubernetes.io/projected/db1b119d-b14a-4fc5-8bd8-cde7f24dc742-kube-api-access-dt7mb\") pod \"perf-node-gather-daemonset-plxfw\" (UID: \"db1b119d-b14a-4fc5-8bd8-cde7f24dc742\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:11:59.952999 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.952968    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xdwz_a96d2294-11ce-403e-9f7f-33d0106bc05e/dns/0.log"
Sep 08 14:11:59.974556 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:11:59.974530    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xdwz_a96d2294-11ce-403e-9f7f-33d0106bc05e/kube-rbac-proxy/0.log"
Sep 08 14:12:00.041748 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.041718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7sl66_272ac4cc-2f4d-45de-b56b-e83b1918116c/dns-node-resolver/0.log"
Sep 08 14:12:00.115403 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.115308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:12:00.415326 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.415286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"]
Sep 08 14:12:00.608665 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.608629    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d98bf66b-t4kh5_95ac8419-8dd3-4b06-bd05-caa2393d1d1a/registry/0.log"
Sep 08 14:12:00.631450 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.631416    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2d8d5_cc1b9c76-e22c-49f3-858f-99ffaa379061/node-ca/0.log"
Sep 08 14:12:00.717481 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.717451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw" event={"ID":"db1b119d-b14a-4fc5-8bd8-cde7f24dc742","Type":"ContainerStarted","Data":"d0fc1a8c601eb76655b6c638e7b68f115994aed0dcd26eb1cbca58acb4a1ce36"}
Sep 08 14:12:00.717669 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.717504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw" event={"ID":"db1b119d-b14a-4fc5-8bd8-cde7f24dc742","Type":"ContainerStarted","Data":"5c326d19ee39f7bf40f3a2438b3e038481f502f208098a3fbd008344b2a0dab4"}
Sep 08 14:12:00.717669 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.717589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:12:00.736301 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:00.736254    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw" podStartSLOduration=1.736239351 podStartE2EDuration="1.736239351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:00.734988697 +0000 UTC m=+3316.685642651" watchObservedRunningTime="2026-09-08 14:12:00.736239351 +0000 UTC m=+3316.686893302"
Sep 08 14:12:01.767855 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:01.767823    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hrgl_2d95a0cd-2074-4446-97db-45ae6a01e3bf/serve-healthcheck-canary/0.log"
Sep 08 14:12:02.203253 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:02.203222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ls8l_eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d/kube-rbac-proxy/0.log"
Sep 08 14:12:02.226594 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:02.226567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ls8l_eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d/exporter/0.log"
Sep 08 14:12:02.251498 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:02.251461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ls8l_eaf8b852-d325-4d6e-9ff1-e4dc3b9c6c1d/extractor/0.log"
Sep 08 14:12:04.890235 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:04.890196    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6f9bbc7b7c-wd56h_1afae64f-7fc3-409a-9408-80bf2cd99ada/manager/0.log"
Sep 08 14:12:04.910741 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:04.910717    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-x8pjd_3d37ad32-b8cd-45d2-8875-5d329a73ec07/server/0.log"
Sep 08 14:12:05.457643 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.457601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j5p7m_c6b91ae4-b98b-422b-bfdd-1d0b020e01b9/s3-init/0.log"
Sep 08 14:12:05.474314 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.474280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j5p7m_c6b91ae4-b98b-422b-bfdd-1d0b020e01b9/download-hf-model/0.log"
Sep 08 14:12:05.511158 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.511127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6ldf9_f6ea7b35-3f02-4031-8956-b2461782a408/s3-tls-init-custom/0.log"
Sep 08 14:12:05.528727 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.528703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6ldf9_f6ea7b35-3f02-4031-8956-b2461782a408/download-hf-model/0.log"
Sep 08 14:12:05.562206 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.562156    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-tqs7n_392ae552-d454-4599-8d6a-b88a306b77c2/s3-tls-init-serving/0.log"
Sep 08 14:12:05.577318 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.577289    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-tqs7n_392ae552-d454-4599-8d6a-b88a306b77c2/download-hf-model/0.log"
Sep 08 14:12:05.605627 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.605601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-hqctt_e05c3d3d-5b29-43a7-9bcc-86bf2e3db6ec/seaweedfs/0.log"
Sep 08 14:12:05.654343 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:05.654317    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-sth9g_5f0c840e-b02a-45d8-b7dc-d9d99183a43c/seaweedfs-tls-serving/0.log"
Sep 08 14:12:06.734185 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:06.734143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-plxfw"
Sep 08 14:12:09.816922 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:09.816897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-j6pmg_daca9765-01e0-4542-beb8-07a507867daf/migrator/0.log"
Sep 08 14:12:09.837408 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:09.837347    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-j6pmg_daca9765-01e0-4542-beb8-07a507867daf/graceful-termination/0.log"
Sep 08 14:12:10.155380 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:10.155300    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-qnzzr_ea5563ed-b472-4c08-8ecc-f61b5cd573ee/kube-storage-version-migrator-operator/1.log"
Sep 08 14:12:10.157807 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:10.157754    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-qnzzr_ea5563ed-b472-4c08-8ecc-f61b5cd573ee/kube-storage-version-migrator-operator/0.log"
Sep 08 14:12:11.241842 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.241805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/kube-multus-additional-cni-plugins/0.log"
Sep 08 14:12:11.263080 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.263047    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/egress-router-binary-copy/0.log"
Sep 08 14:12:11.284195 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.284151    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/cni-plugins/0.log"
Sep 08 14:12:11.306088 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.306058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/bond-cni-plugin/0.log"
Sep 08 14:12:11.327072 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.327036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/routeoverride-cni/0.log"
Sep 08 14:12:11.349253 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.349218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/whereabouts-cni-bincopy/0.log"
Sep 08 14:12:11.373573 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.373539    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfl22_dcabf8af-b206-4d16-b396-8a875b739614/whereabouts-cni/0.log"
Sep 08 14:12:11.694076 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.693995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x52hp_684f95d3-ca3e-45fe-8dad-9487ee06e49d/kube-multus/0.log"
Sep 08 14:12:11.721623 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.721591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q54b_05fa069d-7b89-4ce6-823f-613bfb31539a/network-metrics-daemon/0.log"
Sep 08 14:12:11.738983 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:11.738957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q54b_05fa069d-7b89-4ce6-823f-613bfb31539a/kube-rbac-proxy/0.log"
Sep 08 14:12:13.000138 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.000109    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-controller/0.log"
Sep 08 14:12:13.018089 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.018064    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/1.log"
Sep 08 14:12:13.018447 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.018426    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovn-acl-logging/0.log"
Sep 08 14:12:13.040766 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.040741    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/kube-rbac-proxy-node/0.log"
Sep 08 14:12:13.063796 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.063774    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 14:12:13.084643 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.084619    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/northd/0.log"
Sep 08 14:12:13.104975 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.104953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/nbdb/0.log"
Sep 08 14:12:13.126616 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.126591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/sbdb/0.log"
Sep 08 14:12:13.194923 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:13.194888    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lv5hb_bb50200c-d163-4934-9370-389b297bf6b3/ovnkube-controller/0.log"
Sep 08 14:12:14.177904 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:14.177867    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-k5hjh_c10475ca-573e-4e8c-b618-c0f761e5e1ac/check-endpoints/0.log"
Sep 08 14:12:14.228476 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:14.228451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8cbrp_c09eb737-001f-4c00-bd76-492f842e8980/network-check-target-container/0.log"
Sep 08 14:12:15.201522 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:15.201494    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fqt4r_5fcd4a12-9080-4095-9e1f-28c1107d6de5/iptables-alerter/0.log"
Sep 08 14:12:15.867532 ip-10-0-138-123 kubenswrapper[2580]: I0908 14:12:15.867501    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-5p97k_228911e2-1e89-4968-9b8f-bb2d6f597d2e/tuned/0.log"