Sep 19 08:20:35.566098 ip-10-0-136-228 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:35.950225 ip-10-0-136-228 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:35.950225 ip-10-0-136-228 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:35.950225 ip-10-0-136-228 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:35.950225 ip-10-0-136-228 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:35.950225 ip-10-0-136-228 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:35.950941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.950309    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:35.954118 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954088    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.954118 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954118    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.954118 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954122    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.954118 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954126    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.954118 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954130    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954133    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954136    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954139    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954142    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954144    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954150    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954153    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954156    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954159    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954161    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954164    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954168    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954172    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954175    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954178    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954181    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954184    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954187    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954189    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.954264 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954192    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954195    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954197    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954200    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954203    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954205    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954208    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954211    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954213    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954216    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954219    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954221    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954224    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954226    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954230    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954233    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954236    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954240    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954244    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.954748 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954248    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954252    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954255    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954257    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954261    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954265    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954269    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954272    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954275    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954278    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954282    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954284    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954287    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954290    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954293    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954295    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954298    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954301    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954304    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.955246 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954308    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954310    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954313    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954316    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954318    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954321    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954324    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954326    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954330    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954333    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954336    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954339    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954341    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954344    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954348    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954351    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954354    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954356    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954359    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954362    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.955788 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954364    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954368    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954371    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954373    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954376    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954378    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954382    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954384    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954387    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954389    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954392    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954395    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954398    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954400    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954403    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954406    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.954408    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955128    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955135    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955138    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.956511 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955141    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955144    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955147    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955150    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955152    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955155    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955158    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955161    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955164    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955167    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955169    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955172    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955175    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955177    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955180    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955183    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955185    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955188    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955190    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955193    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955195    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.957075 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955198    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955201    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955204    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955207    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955210    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955213    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955215    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955218    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955221    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955223    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955226    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955229    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955232    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955237    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955241    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955244    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955247    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955250    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955254    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955257    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.957617 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955261    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955264    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955267    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955269    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955272    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955275    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955278    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955281    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955284    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955287    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955290    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955293    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955295    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955298    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955300    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955303    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955306    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955309    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955312    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.958116 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955314    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955317    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955319    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955322    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955325    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955327    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955331    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955334    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955336    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955339    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955341    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955344    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955347    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955351    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955354    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955357    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955359    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955362    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955364    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955367    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.958629 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955369    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955373    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955376    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955378    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955381    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955383    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955386    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955388    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955392    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955396    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955399    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955401    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955404    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955407    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955411    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.955415    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959046    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959059    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959062    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.959294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959066    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959069    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959074    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959078    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959081    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959085    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959088    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959091    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959094    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959096    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959099    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959102    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959105    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959108    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959111    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959114    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959116    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959119    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959121    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959124    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.959826 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959127    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959130    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959132    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959140    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959143    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959146    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959149    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959152    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959155    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959158    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959161    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959164    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959167    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959170    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959173    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959176    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959178    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959181    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959184    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.960327 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959188    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959191    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959194    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959196    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959199    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959202    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959204    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959207    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959211    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959213    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959216    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959219    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959222    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959224    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959227    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959231    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959236    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959240    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959244    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.960800 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959247    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959250    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959254    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959257    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959260    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959263    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959266    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959269    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959272    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959275    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959278    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959281    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959283    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959286    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959288    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959291    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959294    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959297    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959299    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959303    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.961317 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959305    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959308    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959312    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959315    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959319    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959321    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959324    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959326    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959329    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959331    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959334    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959337    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959339    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959342    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959344    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959348    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959351    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.959354    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959423    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959462    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:35.961837 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959469    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959473    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959477    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959481    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959485    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959490    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959493    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959496    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959500    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959503    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959506    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959509    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959512    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959515    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959518    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959522    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959527    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959529    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959533    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959536    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959539    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959543    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959546    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:35.962341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959549    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959552    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959555    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959558    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959561    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959565    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959568    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959572    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959575    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959578    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959581    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959584    2566 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959587    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959592    2566 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959595    2566 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959598    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959602    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959604    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959608    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959611    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959614    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959617    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959620    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959622    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959625    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:35.962923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959629    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959633    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959636    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959639    2566 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959642    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959645    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959648    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959652    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959654    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959658    2566 flags.go:64] FLAG: --help="false"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959660    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-136-228.ec2.internal"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959664    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959667    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959670    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959674    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959677    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959680    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959683    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959686    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959689    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959692    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959695    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959697    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959700    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:35.963526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959703    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959706    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959709    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959712    2566 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959714    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959717    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959720    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959726    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959729    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959732    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959735    2566 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959738    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959741    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959744    2566 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959747    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959752    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959755    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959762    2566 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959780    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959784    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959787    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959791    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959794    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959797    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959800    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:35.964132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959807    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959810    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959813    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959816    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959819    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959825    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959828    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959832    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959835    2566 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959839    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959842    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02bf94f05d527174a"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959845    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959848    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959851    2566 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959854    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959857    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959860    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959864    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959867    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959873    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959876    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959879    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959882    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959885    2566 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959888    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:35.964864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959891    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959894    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959897    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959900    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959903    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959907    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959910    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959913    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959916    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959919    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959922    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959925    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959928    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959931    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959936    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959939    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959942    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959947    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959949    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959952    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959955    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959958    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959961    2566 flags.go:64] FLAG: --v="2"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959965    2566 flags.go:64] FLAG: --version="false"
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959969    2566 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:35.965494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959974    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.959979    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960092    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960095    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960099    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960102    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960104    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960107    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960110    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960113    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960116    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960119    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960121    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960124    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960127    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960129    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960132    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960135    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960140    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.966184 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960143    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960147    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960150    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960153    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960156    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960159    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960162    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960164    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960167    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960169    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960172    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960175    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960178    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960180    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960185    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960188    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960191    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960194    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960197    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960200    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.966658 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960203    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960206    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960210    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960213    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960216    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960219    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960222    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960224    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960227    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960230    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960233    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960235    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960238    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960241    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960244    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960247    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960250    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960253    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960255    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960258    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.967188 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960261    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960266    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960269    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960271    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960274    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960278    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960281    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960284    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960287    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960290    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960292    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960295    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960298    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960302    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960304    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960307    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960309    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960312    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960315    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.967715 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960317    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960320    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960323    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960325    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960328    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960331    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960334    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960337    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960339    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960344    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960348    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960351    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960353    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960356    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960359    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960362    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960364    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960367    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960371    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.968233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960375    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.968743 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960379    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.968743 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.960382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.968743 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.960886    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:35.969023 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969001    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:35.969023 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969023    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:35.969083 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969038    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:35.969083 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969044    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969098    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969103    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969107    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969110    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969113    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969116    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969119    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969122    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969125    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969128    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969131    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969134    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969139    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969143    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.969140 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969147    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969150    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969154    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969157    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969160    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969162    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969165    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969168    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969170    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969173    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969176    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969180    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969183    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969186    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969189    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969191    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969194    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969197    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969199    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969202    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.969533 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969204    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969207    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969210    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969212    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969215    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969218    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969220    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969223    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969226    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969228    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969231    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969233    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969236    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969239    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969243    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969245    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969248    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969251    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969254    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.970112 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969258    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969262    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969265    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969268    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969271    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969274    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969277    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969280    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969283    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969285    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969288    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969291    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969294    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969296    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969299    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969302    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969304    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969307    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969310    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.970583 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969312    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969316    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969319    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969321    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969325    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969329    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969333    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969336    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969339    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969342    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969345    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969348    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969351    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969354    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969357    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969359    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969362    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969365    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969367    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.971056 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969386    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969390    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969393    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969397    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969400    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969410    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969413    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969415    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969421    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969545    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969550    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969553    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969556    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969560    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969563    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:35.971513 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969566    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969570    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969572    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969576    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969579    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969582    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969586    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969588    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969591    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969594    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969597    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969600    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969602    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969605    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969608    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969610    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969613    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969616    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969622    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969625    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:35.971893 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969628    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969630    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969632    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969635    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969638    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969640    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969643    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969645    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969648    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969650    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969653    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969656    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969658    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969661    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969663    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969672    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969676    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969680    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969683    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:35.972385 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969685    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969688    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969691    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969694    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969697    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969700    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969702    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969705    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969707    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969711    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969715    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969718    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969725    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969729    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969732    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969735    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969738    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969740    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969743    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:35.972861 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969746    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969748    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969751    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969753    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969756    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969759    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969761    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969764    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969780    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969784    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969787    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969790    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969793    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969796    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969799    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969801    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969804    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969806    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969809    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969812    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:35.973371 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969815    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969818    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969820    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969823    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969825    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969828    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969831    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969834    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969837    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969839    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969842    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969844    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969847    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969849    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:35.969852    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:35.973928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969857    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:35.974344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.969973    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:35.974344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.972806    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:35.974344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.973711    2566 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:35.974344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.973829    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:35.974496 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.974425    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:35.976058 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.976040    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:35.977215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.977196    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:35.987982 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.987962    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:35.990861 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.990841    2566 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:35.991712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.991698    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:35.995180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.995157    2566 fs.go:135] Filesystem UUIDs: map[5554b3df-90c4-4250-a59d-c0aff5edb838:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bbd0bdee-8ea1-4576-9b54-b9906eea902a:/dev/nvme0n1p3]
Sep 19 08:20:35.995246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:35.995178    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:36.000244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.000091    2566 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:35.998869123 +0000 UTC m=+0.319540701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122016 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217bb3185472b704932468d8a60980 SystemUUID:ec217bb3-1854-72b7-0493-2468d8a60980 BootID:f5026fc6-0682-44de-a755-b3d583dbcacf Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:f9:74:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:f9:74:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ff:b6:dc:f1:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:36.000244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.000230    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:36.000409 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.000322    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:36.003527 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.003498    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:36.003676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.003527    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:36.004217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.004201    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:36.004217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.004218    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:36.004299 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.004226    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:36.005005 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.004992    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:36.005571 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.005552    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:36.006572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.006561    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:36.010115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010100    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:36.010271 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010259    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:36.010320 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010299    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:36.010320 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010315    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:36.010405 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010333    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:36.010405 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010348    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:36.010405 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010361    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:36.010755 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010735    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:36.010876 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.010780    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:36.013541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.013521    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:36.015226 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.015211    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:36.015284 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.015242    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:36.016145 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016126    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:36.016145 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016146    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:36.016289 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016154    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:36.016289 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016160    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:36.016289 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016168    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:36.016289 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016176    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:36.016495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016478    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:36.016495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016491    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:36.016495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016499    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:36.016650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016504    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:36.016650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016513    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:36.016650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.016523    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:36.018090 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.018078    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:36.018090 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.018088    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:36.021958 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.021937    2566 server.go:1268] "Started kubelet"
Sep 19 08:20:36.022044 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.021962    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:36.022044 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.022006    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:36.022044 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.022006    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:36.022174 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.022082    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:36.022174 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.022076    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:36.022174 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.022144    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:36.022907 ip-10-0-136-228 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:36.024215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.024201    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:36.026628 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.026600    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:36.028030 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.028011    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:36.030601 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.029747    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-228.ec2.internal.18d6ab1b61ef9d08  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-228.ec2.internal,UID:ip-10-0-136-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-228.ec2.internal,},FirstTimestamp:2026-09-19 08:20:36.021910792 +0000 UTC m=+0.342582370,LastTimestamp:2026-09-19 08:20:36.021910792 +0000 UTC m=+0.342582370,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-228.ec2.internal,}"
Sep 19 08:20:36.033352 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.033336    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:36.033475 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.033359    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:36.033578 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.033565    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.033968 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.033949    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:36.033968 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.033970    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:36.034091 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.034068    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:36.034142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.034117    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:36.034142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.034123    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:36.035676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035660    2566 factory.go:153] Registering CRI-O factory
Sep 19 08:20:36.035791 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035740    2566 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:36.035847 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.035780    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 08:20:36.035847 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035825    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:36.035847 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035839    2566 factory.go:55] Registering systemd factory
Sep 19 08:20:36.035847 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035847    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:36.036042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035864    2566 factory.go:103] Registering Raw factory
Sep 19 08:20:36.036042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035842    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77d92"
Sep 19 08:20:36.036042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.035880    2566 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:36.036042 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.035972    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 08:20:36.036327 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.036312    2566 manager.go:349] Starting recovery of all containers
Sep 19 08:20:36.042293 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.042268    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77d92"
Sep 19 08:20:36.046724 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.046707    2566 manager.go:354] Recovery completed
Sep 19 08:20:36.051442 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.051429    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.053614 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.053592    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.053716 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.053631    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.053716 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.053642    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.054205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054188    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:36.054205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054203    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:36.054325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054206    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.054325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054222    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:36.054891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054876    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.054954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054901    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.054954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.054912    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.056204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.056191    2566 policy_none.go:49] "None policy: Start"
Sep 19 08:20:36.056260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.056209    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:36.056260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.056223    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:36.058270 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.058099    2566 policy_none.go:47] "Start"
Sep 19 08:20:36.102589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.102557    2566 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.102617    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.102632    2566 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.103092    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.103106    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.103212    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.103365    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.103375    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.103891    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:36.112481 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.103932    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.174005 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.173966    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:36.175196 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.175179    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:36.175292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.175232    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:36.175292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.175258    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:36.175394 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.175360    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 08:20:36.177637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.177617    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:36.203507 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.203409    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.204549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.204532    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.204663 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.204569    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.204663 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.204583    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.204663 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.204614    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.210884 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.210868    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.210947 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.210890    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-228.ec2.internal\": node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.231689 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.231657    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.275514 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.275468    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"]
Sep 19 08:20:36.275708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.275579    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.277621 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.277604    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.277689 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.277642    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.277689 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.277656    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.280018 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.280004    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.280140 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.280125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.280196 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.280163    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.281434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281418    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.281434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281427    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.281568 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281444    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.281568 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281460    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.281568 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281463    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.281568 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.281474    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.283846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.283830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.283925 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.283854    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:36.284668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.284652    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:36.284749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.284677    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:36.284749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.284694    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:36.302412 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.302393    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.305947 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.305929    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.331887 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.331857    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.432913 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.432831    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.436132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.436114    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.436184 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.436137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.436184 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.436155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2692f115a698526b3ec4e4ebc7f1819f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"2692f115a698526b3ec4e4ebc7f1819f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.533065 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.532981    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.536283 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.536339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.536339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2692f115a698526b3ec4e4ebc7f1819f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"2692f115a698526b3ec4e4ebc7f1819f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.536400 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.536433 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/990e8c6cbd2b769983e6f96dbfebc6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"990e8c6cbd2b769983e6f96dbfebc6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.536465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.536379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2692f115a698526b3ec4e4ebc7f1819f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"2692f115a698526b3ec4e4ebc7f1819f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.605479 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.605414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.609603 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.609583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:36.633829 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.633800    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.734350 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.734304    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.834806 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.834699    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.935183 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:36.935146    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:36.973496 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.973469    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:36.974104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:36.973615    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:37.035568 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:37.035540    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:37.047444 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.047404    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:36 +0000 UTC" deadline="2028-05-28 03:05:11.694989552 +0000 UTC"
Sep 19 08:20:37.047444 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.047436    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14802h44m34.64755773s"
Sep 19 08:20:37.135854 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:37.135777    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:37.236256 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:37.236230    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 19 08:20:37.236782 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.236743    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:37.238507 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:37.238476    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2692f115a698526b3ec4e4ebc7f1819f.slice/crio-ab103ff28fb3d1a71f23c41eaf1e5a41966af79c2d18a611cf20035dceca3aa5 WatchSource:0}: Error finding container ab103ff28fb3d1a71f23c41eaf1e5a41966af79c2d18a611cf20035dceca3aa5: Status 404 returned error can't find the container with id ab103ff28fb3d1a71f23c41eaf1e5a41966af79c2d18a611cf20035dceca3aa5
Sep 19 08:20:37.238983 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:37.238964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990e8c6cbd2b769983e6f96dbfebc6b4.slice/crio-6590e7c1a6ee91b1580c1fc5d6b1385bb2c473f51fdae20e1edc573523888c21 WatchSource:0}: Error finding container 6590e7c1a6ee91b1580c1fc5d6b1385bb2c473f51fdae20e1edc573523888c21: Status 404 returned error can't find the container with id 6590e7c1a6ee91b1580c1fc5d6b1385bb2c473f51fdae20e1edc573523888c21
Sep 19 08:20:37.243466 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.243451    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:37.331083 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.331042    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:37.334175 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.334152    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:37.346318 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.346293    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:37.347967 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.347952    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 19 08:20:37.359008 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.358987    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:37.518297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.518219    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:37.898265 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:37.898183    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:38.011298 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.011259    2566 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:38.023012 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.022980    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:38.023531 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.023502    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal","openshift-multus/multus-99ch6","openshift-multus/multus-additional-cni-plugins-8mlvx","openshift-multus/network-metrics-daemon-c4vdw","openshift-network-operator/iptables-alerter-cm2b2","kube-system/global-pull-secret-syncer-h7m8x","kube-system/konnectivity-agent-56xxt","kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal","openshift-image-registry/node-ca-6hdz7","openshift-network-diagnostics/network-check-target-dtq79","openshift-ovn-kubernetes/ovnkube-node-clzwt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj","openshift-cluster-node-tuning-operator/tuned-vbccp","openshift-dns/node-resolver-vf9n9"]
Sep 19 08:20:38.026160 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.026134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.028326 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.028305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.028920 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.028898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:38.029043 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.029025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x468b\""
Sep 19 08:20:38.029111 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.029031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:38.030651 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.030459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.031084 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.031063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.031212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.031191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:38.031279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.031258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fhlwh\""
Sep 19 08:20:38.031330 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.031302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:38.031611 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.031593    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.033554 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.033487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.033665 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.033628    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:38.033971 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.033950    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:38.034058 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.033963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ghbb8\""
Sep 19 08:20:38.034294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.034277    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:38.034493 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.034470    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:38.035836 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.035819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.038571 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.038548    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.038668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.038623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.038763 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.038747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5c2vz\""
Sep 19 08:20:38.038838 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.038817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.040728 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.040712    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.041866 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.041826    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:38.041991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.041973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.042148 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.042131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skrk7\""
Sep 19 08:20:38.042219 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.042175    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.043352 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.043127    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.044060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.044005    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:38.045180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.045157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf05c100-c217-44d7-a410-b5e695f03388-agent-certs\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.045878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.045329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b5575\""
Sep 19 08:20:38.045878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.045504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.045878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.045703    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.046099 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046056    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.046317 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whg67\""
Sep 19 08:20:38.046378 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.046554 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046535    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.046656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-bin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.046715 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-kubelet\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.046758 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-hostroot\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.046890 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.046890 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-k8s-cni-cncf-io\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.046890 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-netns\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.046890 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-system-cni-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-cnibin\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-multus-certs\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.046984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1704028-5d63-44ab-b038-4867a45a8896-tmp-dir\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-system-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-conf-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-multus-daemon-config\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jsk\" (UniqueName: \"kubernetes.io/projected/2529d58c-8035-4da8-9411-00283e9e465d-kube-api-access-t7jsk\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nph\" (UniqueName: \"kubernetes.io/projected/7d2b547b-9039-4c47-851b-9c6d065e56d8-kube-api-access-x8nph\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1704028-5d63-44ab-b038-4867a45a8896-hosts-file\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-etc-kubernetes\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf05c100-c217-44d7-a410-b5e695f03388-konnectivity-ca\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-cnibin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-cni-binary-copy\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-multus\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047462    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcl9\" (UniqueName: \"kubernetes.io/projected/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-kube-api-access-rgcl9\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-socket-dir-parent\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.047601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.047563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnk8\" (UniqueName: \"kubernetes.io/projected/f1704028-5d63-44ab-b038-4867a45a8896-kube-api-access-6gnk8\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.048174 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.048153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.048796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.048535    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.048796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.048731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:38.048796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.048735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:38.048971 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.048809    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:38.049580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.049560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nfld6\""
Sep 19 08:20:38.054168 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.054150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.055999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.055966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:38.056482 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.056465    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:38.056860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.056692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:38.056860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.056820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.057030 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.056878    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:38.057030 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.056914    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:38.057141 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.057122    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f4fzs\""
Sep 19 08:20:38.057355 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.057209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:38.058281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.058258    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:38.069863 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.069832    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rw9d2"
Sep 19 08:20:38.080626 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.080449    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rw9d2"
Sep 19 08:20:38.135205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.135175    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:38.148410 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.148410 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-systemd\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.148410 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-kubelet\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-etc-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvln\" (UniqueName: \"kubernetes.io/projected/3921dcbc-5bec-4b64-a499-2a9340efaf3e-kube-api-access-bcvln\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.148524    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-multus\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.148589    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.648572464 +0000 UTC m=+2.969244031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-multus\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcl9\" (UniqueName: \"kubernetes.io/projected/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-kube-api-access-rgcl9\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.148669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921dcbc-5bec-4b64-a499-2a9340efaf3e-iptables-alerter-script\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-kubelet-config\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-config\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-device-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-kubelet\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-tuned\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-kubelet\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-var-lib-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.148988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-node-log\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-system-cni-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-systemd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-systemd-units\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-system-cni-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7x5\" (UniqueName: \"kubernetes.io/projected/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-kube-api-access-rd7x5\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-conf\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-var-lib-kubelet\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovn-node-metrics-cert\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsbq\" (UniqueName: \"kubernetes.io/projected/d861b7f9-efdf-4245-b57b-9ac9973cca19-kube-api-access-cvsbq\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chb9m\" (UniqueName: \"kubernetes.io/projected/67a02ae7-ea30-4518-b05c-236b3630efe5-kube-api-access-chb9m\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-multus-daemon-config\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nph\" (UniqueName: \"kubernetes.io/projected/7d2b547b-9039-4c47-851b-9c6d065e56d8-kube-api-access-x8nph\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1704028-5d63-44ab-b038-4867a45a8896-hosts-file\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.149549 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-log-socket\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1704028-5d63-44ab-b038-4867a45a8896-hosts-file\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-script-lib\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-etc-kubernetes\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-dbus\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-modprobe-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-slash\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-socket-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-etc-kubernetes\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf05c100-c217-44d7-a410-b5e695f03388-konnectivity-ca\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-cnibin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-run\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-cni-binary-copy\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-lib-modules\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-host\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjz45\" (UniqueName: \"kubernetes.io/projected/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-kube-api-access-jjz45\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-multus-daemon-config\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.149997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-bin\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-cnibin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-socket-dir-parent\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnk8\" (UniqueName: \"kubernetes.io/projected/f1704028-5d63-44ab-b038-4867a45a8896-kube-api-access-6gnk8\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-serviceca\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-socket-dir-parent\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysconfig\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-netd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-registration-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf05c100-c217-44d7-a410-b5e695f03388-konnectivity-ca\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf05c100-c217-44d7-a410-b5e695f03388-agent-certs\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-bin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-hostroot\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2529d58c-8035-4da8-9411-00283e9e465d-cni-binary-copy\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.150977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-host\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-var-lib-cni-bin\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-sys\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-hostroot\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-k8s-cni-cncf-io\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-netns\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-cnibin\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-netns\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-k8s-cni-cncf-io\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-env-overrides\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-sys-fs\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-multus-certs\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.151789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-cnibin\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150763    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1704028-5d63-44ab-b038-4867a45a8896-tmp-dir\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d2b547b-9039-4c47-851b-9c6d065e56d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-tmp\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-netns\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-host-run-multus-certs\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-ovn\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921dcbc-5bec-4b64-a499-2a9340efaf3e-host-slash\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-system-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.150979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-conf-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jsk\" (UniqueName: \"kubernetes.io/projected/2529d58c-8035-4da8-9411-00283e9e465d-kube-api-access-t7jsk\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1704028-5d63-44ab-b038-4867a45a8896-tmp-dir\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.152572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-kubernetes\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.153478 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-multus-conf-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.153478 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2529d58c-8035-4da8-9411-00283e9e465d-system-cni-dir\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.153478 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.151333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d2b547b-9039-4c47-851b-9c6d065e56d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.154893 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.154874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf05c100-c217-44d7-a410-b5e695f03388-agent-certs\") pod \"konnectivity-agent-56xxt\" (UID: \"bf05c100-c217-44d7-a410-b5e695f03388\") " pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.160534 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.160514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nph\" (UniqueName: \"kubernetes.io/projected/7d2b547b-9039-4c47-851b-9c6d065e56d8-kube-api-access-x8nph\") pod \"multus-additional-cni-plugins-8mlvx\" (UID: \"7d2b547b-9039-4c47-851b-9c6d065e56d8\") " pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.161786 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.161744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcl9\" (UniqueName: \"kubernetes.io/projected/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-kube-api-access-rgcl9\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.162437 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.162412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jsk\" (UniqueName: \"kubernetes.io/projected/2529d58c-8035-4da8-9411-00283e9e465d-kube-api-access-t7jsk\") pod \"multus-99ch6\" (UID: \"2529d58c-8035-4da8-9411-00283e9e465d\") " pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.162543 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.162488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnk8\" (UniqueName: \"kubernetes.io/projected/f1704028-5d63-44ab-b038-4867a45a8896-kube-api-access-6gnk8\") pod \"node-resolver-vf9n9\" (UID: \"f1704028-5d63-44ab-b038-4867a45a8896\") " pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.180074 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.180026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"990e8c6cbd2b769983e6f96dbfebc6b4","Type":"ContainerStarted","Data":"6590e7c1a6ee91b1580c1fc5d6b1385bb2c473f51fdae20e1edc573523888c21"}
Sep 19 08:20:38.184132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.184091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"2692f115a698526b3ec4e4ebc7f1819f","Type":"ContainerStarted","Data":"ab103ff28fb3d1a71f23c41eaf1e5a41966af79c2d18a611cf20035dceca3aa5"}
Sep 19 08:20:38.251671 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.251671 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-env-overrides\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-sys-fs\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-tmp\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-netns\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-ovn\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921dcbc-5bec-4b64-a499-2a9340efaf3e-host-slash\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-sys-fs\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921dcbc-5bec-4b64-a499-2a9340efaf3e-host-slash\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-netns\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.251933 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-kubernetes\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-systemd\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.251984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-kubelet\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-etc-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvln\" (UniqueName: \"kubernetes.io/projected/3921dcbc-5bec-4b64-a499-2a9340efaf3e-kube-api-access-bcvln\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921dcbc-5bec-4b64-a499-2a9340efaf3e-iptables-alerter-script\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-kubelet-config\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-config\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-device-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-tuned\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-kubelet\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-var-lib-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-ovn\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-node-log\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-node-log\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-systemd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.252547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-etc-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-systemd-units\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-env-overrides\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7x5\" (UniqueName: \"kubernetes.io/projected/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-kube-api-access-rd7x5\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-kubelet-config\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-conf\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.252422    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-device-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-var-lib-kubelet\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.252478    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.752462471 +0000 UTC m=+3.073134039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-kubernetes\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-systemd\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-conf\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-systemd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysctl-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.253408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-var-lib-kubelet\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-systemd-units\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovn-node-metrics-cert\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsbq\" (UniqueName: \"kubernetes.io/projected/d861b7f9-efdf-4245-b57b-9ac9973cca19-kube-api-access-cvsbq\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-run-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-var-lib-openvswitch\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-config\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chb9m\" (UniqueName: \"kubernetes.io/projected/67a02ae7-ea30-4518-b05c-236b3630efe5-kube-api-access-chb9m\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-log-socket\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-script-lib\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921dcbc-5bec-4b64-a499-2a9340efaf3e-iptables-alerter-script\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.252987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-dbus\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-dbus\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-modprobe-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-slash\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-socket-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.254238 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-run\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-lib-modules\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-modprobe-d\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-host\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-log-socket\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-run-ovn-kubernetes\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-run\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-host\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-lib-modules\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjz45\" (UniqueName: \"kubernetes.io/projected/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-kube-api-access-jjz45\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-bin\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-socket-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-bin\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-slash\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-serviceca\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysconfig\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-netd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-registration-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.255032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-host\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-sys\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-registration-dir\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d861b7f9-efdf-4245-b57b-9ac9973cca19-host-cni-netd\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-host\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-sysconfig\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67a02ae7-ea30-4518-b05c-236b3630efe5-etc-selinux\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovnkube-script-lib\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-sys\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.253953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-serviceca\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.255334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d861b7f9-efdf-4245-b57b-9ac9973cca19-ovn-node-metrics-cert\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.255664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.255393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-etc-tuned\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.256122 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.255883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-tmp\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.264939 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.264914    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:38.264939 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.264942    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:38.265123 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.264967    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:38.265123 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.265044    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.765025875 +0000 UTC m=+3.085697448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:38.266051 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.266020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvln\" (UniqueName: \"kubernetes.io/projected/3921dcbc-5bec-4b64-a499-2a9340efaf3e-kube-api-access-bcvln\") pod \"iptables-alerter-cm2b2\" (UID: \"3921dcbc-5bec-4b64-a499-2a9340efaf3e\") " pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.267551 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.267522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsbq\" (UniqueName: \"kubernetes.io/projected/d861b7f9-efdf-4245-b57b-9ac9973cca19-kube-api-access-cvsbq\") pod \"ovnkube-node-clzwt\" (UID: \"d861b7f9-efdf-4245-b57b-9ac9973cca19\") " pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.267825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.267801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chb9m\" (UniqueName: \"kubernetes.io/projected/67a02ae7-ea30-4518-b05c-236b3630efe5-kube-api-access-chb9m\") pod \"aws-ebs-csi-driver-node-sn5sj\" (UID: \"67a02ae7-ea30-4518-b05c-236b3630efe5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.268347 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.268323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjz45\" (UniqueName: \"kubernetes.io/projected/8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4-kube-api-access-jjz45\") pod \"tuned-vbccp\" (UID: \"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.268563 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.268542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7x5\" (UniqueName: \"kubernetes.io/projected/e93a1df2-a189-4c33-aad1-d17ca2d7cc1d-kube-api-access-rd7x5\") pod \"node-ca-6hdz7\" (UID: \"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d\") " pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.339564 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.339534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:20:38.348921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.348889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-99ch6"
Sep 19 08:20:38.357524 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.357476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8mlvx"
Sep 19 08:20:38.367589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.367565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cm2b2"
Sep 19 08:20:38.376525 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.376500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vf9n9"
Sep 19 08:20:38.383386 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.383357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj"
Sep 19 08:20:38.389212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.389136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hdz7"
Sep 19 08:20:38.395094 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.395072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbccp"
Sep 19 08:20:38.398935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.398882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:20:38.657000 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.656905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:38.657193 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.657062    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.657193 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.657144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.657121326 +0000 UTC m=+3.977792909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.758166 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.758122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:38.758380 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.758278    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.758380 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.758349    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.758332001 +0000 UTC m=+4.079003578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.859298 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:38.859255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:38.859508 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.859482    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:38.859557 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.859517    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:38.859557 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.859533    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:38.859640 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:38.859608    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.859584188 +0000 UTC m=+4.180255763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:39.084245 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.084148    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:38 +0000 UTC" deadline="2028-02-23 20:25:11.741971609 +0000 UTC"
Sep 19 08:20:39.084245 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.084185    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12540h4m32.657792691s"
Sep 19 08:20:39.137294 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.137254    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93a1df2_a189_4c33_aad1_d17ca2d7cc1d.slice/crio-eafdb2d104ed9589f9f73ffce509f8a120d34020ec67272f15f709cb320f606b WatchSource:0}: Error finding container eafdb2d104ed9589f9f73ffce509f8a120d34020ec67272f15f709cb320f606b: Status 404 returned error can't find the container with id eafdb2d104ed9589f9f73ffce509f8a120d34020ec67272f15f709cb320f606b
Sep 19 08:20:39.138242 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.138207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd861b7f9_efdf_4245_b57b_9ac9973cca19.slice/crio-11cd0eae5cb2b7d8f6aac050ce490bf2c55c15e01025460da09c8e4410193888 WatchSource:0}: Error finding container 11cd0eae5cb2b7d8f6aac050ce490bf2c55c15e01025460da09c8e4410193888: Status 404 returned error can't find the container with id 11cd0eae5cb2b7d8f6aac050ce490bf2c55c15e01025460da09c8e4410193888
Sep 19 08:20:39.141663 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.141637    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf05c100_c217_44d7_a410_b5e695f03388.slice/crio-a67f6b7ee4d4e99e4cd8480f54cbe5dacb45b819f41a05382c995bcc6306aa30 WatchSource:0}: Error finding container a67f6b7ee4d4e99e4cd8480f54cbe5dacb45b819f41a05382c995bcc6306aa30: Status 404 returned error can't find the container with id a67f6b7ee4d4e99e4cd8480f54cbe5dacb45b819f41a05382c995bcc6306aa30
Sep 19 08:20:39.144632 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.144610    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3921dcbc_5bec_4b64_a499_2a9340efaf3e.slice/crio-9724d76c39e8ce625f60bcceb10915d65986ad72039defff74f5c2c5b688f71f WatchSource:0}: Error finding container 9724d76c39e8ce625f60bcceb10915d65986ad72039defff74f5c2c5b688f71f: Status 404 returned error can't find the container with id 9724d76c39e8ce625f60bcceb10915d65986ad72039defff74f5c2c5b688f71f
Sep 19 08:20:39.146161 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.146131    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1704028_5d63_44ab_b038_4867a45a8896.slice/crio-8c02f54e19c576380977d77a80b195186def8207b9d2938aad23c1abff55a9ba WatchSource:0}: Error finding container 8c02f54e19c576380977d77a80b195186def8207b9d2938aad23c1abff55a9ba: Status 404 returned error can't find the container with id 8c02f54e19c576380977d77a80b195186def8207b9d2938aad23c1abff55a9ba
Sep 19 08:20:39.147173 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.147158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a02ae7_ea30_4518_b05c_236b3630efe5.slice/crio-cca28de43923973543bb314e600e2d9d8c7328ee2e8f72973fe1c3de936ed631 WatchSource:0}: Error finding container cca28de43923973543bb314e600e2d9d8c7328ee2e8f72973fe1c3de936ed631: Status 404 returned error can't find the container with id cca28de43923973543bb314e600e2d9d8c7328ee2e8f72973fe1c3de936ed631
Sep 19 08:20:39.148268 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.148223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea52622_a3a6_482d_9a0e_dcb6df7a8bf4.slice/crio-fc06d4dfe9c27286200bc84e2ecc8fdca8d043a87ad9227755e7c4cd9eb4196a WatchSource:0}: Error finding container fc06d4dfe9c27286200bc84e2ecc8fdca8d043a87ad9227755e7c4cd9eb4196a: Status 404 returned error can't find the container with id fc06d4dfe9c27286200bc84e2ecc8fdca8d043a87ad9227755e7c4cd9eb4196a
Sep 19 08:20:39.149313 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.149272    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2529d58c_8035_4da8_9411_00283e9e465d.slice/crio-3c0f6558b506eb9216bc0d23f8bb6071823ce00589084c025c7242fe74fc0d49 WatchSource:0}: Error finding container 3c0f6558b506eb9216bc0d23f8bb6071823ce00589084c025c7242fe74fc0d49: Status 404 returned error can't find the container with id 3c0f6558b506eb9216bc0d23f8bb6071823ce00589084c025c7242fe74fc0d49
Sep 19 08:20:39.150127 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:20:39.150107    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2b547b_9039_4c47_851b_9c6d065e56d8.slice/crio-5dc8fb450b80054047023ee41a2230a2fdfe5b8922a5f0dccda55110eda5025d WatchSource:0}: Error finding container 5dc8fb450b80054047023ee41a2230a2fdfe5b8922a5f0dccda55110eda5025d: Status 404 returned error can't find the container with id 5dc8fb450b80054047023ee41a2230a2fdfe5b8922a5f0dccda55110eda5025d
Sep 19 08:20:39.188335 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.188059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hdz7" event={"ID":"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d","Type":"ContainerStarted","Data":"eafdb2d104ed9589f9f73ffce509f8a120d34020ec67272f15f709cb320f606b"}
Sep 19 08:20:39.189631 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.189600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerStarted","Data":"5dc8fb450b80054047023ee41a2230a2fdfe5b8922a5f0dccda55110eda5025d"}
Sep 19 08:20:39.190845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.190818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-99ch6" event={"ID":"2529d58c-8035-4da8-9411-00283e9e465d","Type":"ContainerStarted","Data":"3c0f6558b506eb9216bc0d23f8bb6071823ce00589084c025c7242fe74fc0d49"}
Sep 19 08:20:39.194846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.194809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbccp" event={"ID":"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4","Type":"ContainerStarted","Data":"fc06d4dfe9c27286200bc84e2ecc8fdca8d043a87ad9227755e7c4cd9eb4196a"}
Sep 19 08:20:39.197111 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.197082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"11cd0eae5cb2b7d8f6aac050ce490bf2c55c15e01025460da09c8e4410193888"}
Sep 19 08:20:39.200642 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.200618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj" event={"ID":"67a02ae7-ea30-4518-b05c-236b3630efe5","Type":"ContainerStarted","Data":"cca28de43923973543bb314e600e2d9d8c7328ee2e8f72973fe1c3de936ed631"}
Sep 19 08:20:39.201885 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.201857    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cm2b2" event={"ID":"3921dcbc-5bec-4b64-a499-2a9340efaf3e","Type":"ContainerStarted","Data":"9724d76c39e8ce625f60bcceb10915d65986ad72039defff74f5c2c5b688f71f"}
Sep 19 08:20:39.203000 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.202979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vf9n9" event={"ID":"f1704028-5d63-44ab-b038-4867a45a8896","Type":"ContainerStarted","Data":"8c02f54e19c576380977d77a80b195186def8207b9d2938aad23c1abff55a9ba"}
Sep 19 08:20:39.204104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.204082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56xxt" event={"ID":"bf05c100-c217-44d7-a410-b5e695f03388","Type":"ContainerStarted","Data":"a67f6b7ee4d4e99e4cd8480f54cbe5dacb45b819f41a05382c995bcc6306aa30"}
Sep 19 08:20:39.666358 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.665563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:39.666358 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.665802    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:39.666358 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.665899    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:41.665876848 +0000 UTC m=+5.986548432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:39.766600 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.766559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:39.766817 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.766722    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:39.766817 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.766778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:20:41.766762792 +0000 UTC m=+6.087434359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:39.867873 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:39.867240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:39.867873 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.867479    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:39.867873 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.867505    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:39.867873 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.867520    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:39.867873 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:39.867587    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:41.867568224 +0000 UTC m=+6.188239817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:40.085133 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.085057    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:38 +0000 UTC" deadline="2028-02-17 14:37:26.725733028 +0000 UTC"
Sep 19 08:20:40.085133 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.085130    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12390h16m46.640607651s"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.176984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:40.177157    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.177549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:40.177663    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.177733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:40.177923 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:40.177804    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:40.227704 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.227602    2566 generic.go:356] "Generic (PLEG): container finished" podID="990e8c6cbd2b769983e6f96dbfebc6b4" containerID="e0beed54daa7165aca476373c7905e93d26b1074358e67ce4acfb038f7e8b443" exitCode=0
Sep 19 08:20:40.227998 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.227969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"990e8c6cbd2b769983e6f96dbfebc6b4","Type":"ContainerDied","Data":"e0beed54daa7165aca476373c7905e93d26b1074358e67ce4acfb038f7e8b443"}
Sep 19 08:20:40.247778 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:40.245740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"2692f115a698526b3ec4e4ebc7f1819f","Type":"ContainerStarted","Data":"6abf58fb2683ba5fa81a39b113b44a6971e2c3e87bfa9d8c1d41709f2296ad54"}
Sep 19 08:20:41.289391 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.289346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"990e8c6cbd2b769983e6f96dbfebc6b4","Type":"ContainerStarted","Data":"efa1c7b50e8bd1d22e00738d719ed7dd9ce85b4040fea96501e85b5a9e16c969"}
Sep 19 08:20:41.306071 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.305998    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" podStartSLOduration=3.30597384 podStartE2EDuration="3.30597384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:40.266751243 +0000 UTC m=+4.587422834" watchObservedRunningTime="2026-09-19 08:20:41.30597384 +0000 UTC m=+5.626645432"
Sep 19 08:20:41.306718 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.306677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" podStartSLOduration=3.306663684 podStartE2EDuration="3.306663684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:41.306164562 +0000 UTC m=+5.626836145" watchObservedRunningTime="2026-09-19 08:20:41.306663684 +0000 UTC m=+5.627335275"
Sep 19 08:20:41.688633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.688536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:41.688826 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.688703    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:41.688826 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.688773    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:45.688754362 +0000 UTC m=+10.009425945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:41.790413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.789497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:41.790413 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.789739    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:41.790413 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.789829    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:20:45.789793327 +0000 UTC m=+10.110464923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:41.891687 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:41.890938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:41.891687 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.891167    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:41.891687 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.891190    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:41.891687 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.891203    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:41.891687 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:41.891271    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:45.891251403 +0000 UTC m=+10.211922974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:42.177800 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:42.177055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:42.177800 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:42.177221    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:42.177800 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:42.177057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:42.177800 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:42.177674    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:42.178355 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:42.178199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:42.178355 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:42.178314    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:44.175807 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:44.175753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:44.176334 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:44.175911    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:44.176413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:44.176346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:44.176487 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:44.176457    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:44.176861 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:44.176660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:44.176861 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:44.176795    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:45.724382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:45.724313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:45.724939 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.724515    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:45.724939 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.724611    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.724583237 +0000 UTC m=+18.045254810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:45.825385 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:45.824738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:45.825385 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.824924    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:45.825385 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.824996    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.824976842 +0000 UTC m=+18.145648415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:45.925632 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:45.925591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:45.925845 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.925783    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:45.925845 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.925833    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:45.925956 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.925847    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:45.925956 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:45.925921    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.925900986 +0000 UTC m=+18.246572569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:46.176573 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:46.176486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:46.176758 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:46.176626    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:46.177085 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:46.177059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:46.177184 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:46.177166    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:46.177246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:46.177219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:46.177311 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:46.177294    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:48.179758 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:48.179727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:48.180196 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:48.179789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:48.180196 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:48.179843    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:48.180196 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:48.179906    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:48.180196 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:48.179961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:48.182787 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:48.181466    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:50.175900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:50.175870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:50.175900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:50.175893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:50.176356 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:50.175907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:50.176356 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:50.176001    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:50.176356 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:50.176073    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:50.176356 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:50.176137    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:52.176290 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:52.176097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:52.176290 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:52.176116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:52.176290 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:52.176097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:52.176290 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:52.176236    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:52.176861 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:52.176307    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:52.176861 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:52.176398    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:53.786713 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:53.786682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:53.787028 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.786786    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.787028 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.786836    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.786820086 +0000 UTC m=+34.107491665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.887778 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:53.887751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:53.887876 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.887856    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.887911 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.887897    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.887886906 +0000 UTC m=+34.208558476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.989047 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:53.988969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:53.989175 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.989153    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:53.989218 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.989181    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:53.989218 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.989193    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:53.989301 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:53.989251    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.989233182 +0000 UTC m=+34.309904764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.175684 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:54.175651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:54.175883 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:54.175650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:54.175883 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:54.175797    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:54.175973 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:54.175911    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:54.175973 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:54.175664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:54.176044 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:54.176010    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:56.176456 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:56.176261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:56.176895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:56.176344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:56.176895 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:56.176538    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:56.176895 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:56.176641    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:56.176895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:56.176367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:56.176895 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:56.176751    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:58.176298 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.176081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:20:58.177062 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.176089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:20:58.177062 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:58.176347    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:20:58.177062 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:58.176397    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:20:58.177062 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.176124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:20:58.177062 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:20:58.176488    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:20:58.344079 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.344051    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="01daab6ae9f885051e5453758f9b9258bbc748067594c448dad122251930f704" exitCode=0
Sep 19 08:20:58.344244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.344131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"01daab6ae9f885051e5453758f9b9258bbc748067594c448dad122251930f704"}
Sep 19 08:20:58.345654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.345631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-99ch6" event={"ID":"2529d58c-8035-4da8-9411-00283e9e465d","Type":"ContainerStarted","Data":"6edee45a4ef8edd544283d18242d477c4e1a2338e85298bbaab65b5212dd9ba0"}
Sep 19 08:20:58.347133 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.347109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbccp" event={"ID":"8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4","Type":"ContainerStarted","Data":"725eb3f41a06df46d2eb8462e51753ecf13aaf77ad122a91a09df1c577249b46"}
Sep 19 08:20:58.350091 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"0935dde6cceb427705b050e36d4a353b9dc2622c1724b4b86ad9d671b0cfd52b"}
Sep 19 08:20:58.350181 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"1739701e3665d3ce1ca62776dee4ff37defe02054fd726627b76b98252711047"}
Sep 19 08:20:58.350229 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"ad33bf3a6284084dbb5c4728abf67d18ebc760024e76b51c1c738e09f80253a8"}
Sep 19 08:20:58.350263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350241    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"44c5e21018191c31c95b6556df8edecb66dd9e6e31196cb2ba4379f62343a089"}
Sep 19 08:20:58.350304 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"bd8cea1ed442c6459c4ca67a9d9dc407f5111bb03adb53a870ae0af008376405"}
Sep 19 08:20:58.350347 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.350317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"ac570725ee5f5a50d45491d0874248b562b17cf37bd88deeef61a67fb961ed4e"}
Sep 19 08:20:58.354394 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.354372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj" event={"ID":"67a02ae7-ea30-4518-b05c-236b3630efe5","Type":"ContainerStarted","Data":"a3610d46d93448c3c10eaf10e01e3e401ac3a5ef046165a91c0e11c3fb274309"}
Sep 19 08:20:58.355690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.355669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vf9n9" event={"ID":"f1704028-5d63-44ab-b038-4867a45a8896","Type":"ContainerStarted","Data":"447dcf7bfb8c77b5aabb6924506c387c33b81d9be3b08812336880ff9626a4fe"}
Sep 19 08:20:58.356981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.356941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56xxt" event={"ID":"bf05c100-c217-44d7-a410-b5e695f03388","Type":"ContainerStarted","Data":"22922fd9d1a44f2185d9867db1cd58516eb4c02eefae4353e7ca4e8f987618ec"}
Sep 19 08:20:58.358311 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.358295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hdz7" event={"ID":"e93a1df2-a189-4c33-aad1-d17ca2d7cc1d","Type":"ContainerStarted","Data":"907cdd5065330fd7c089a17e9616bc0c01e0a0389c6251ba4ab688e0751f05a2"}
Sep 19 08:20:58.385263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.385218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vbccp" podStartSLOduration=4.333428579 podStartE2EDuration="22.385207842s" totalImagesPullingTime="18.051779263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:58.3849461 +0000 UTC m=+22.705617687" watchObservedRunningTime="2026-09-19 08:20:58.385207842 +0000 UTC m=+22.705879429"
Sep 19 08:20:58.419608 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.419571    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-99ch6" podStartSLOduration=4.315308373 podStartE2EDuration="22.419558041s" totalImagesPullingTime="18.104249668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:58.405030743 +0000 UTC m=+22.725702328" watchObservedRunningTime="2026-09-19 08:20:58.419558041 +0000 UTC m=+22.740229629"
Sep 19 08:20:58.420164 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.420141    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vf9n9" podStartSLOduration=4.368786295 podStartE2EDuration="22.420134598s" totalImagesPullingTime="18.051348303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:58.419471521 +0000 UTC m=+22.740143112" watchObservedRunningTime="2026-09-19 08:20:58.420134598 +0000 UTC m=+22.740806185"
Sep 19 08:20:58.434311 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.434251    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6hdz7" podStartSLOduration=4.374042713 podStartE2EDuration="22.434241231s" totalImagesPullingTime="18.060198518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:58.434139494 +0000 UTC m=+22.754811083" watchObservedRunningTime="2026-09-19 08:20:58.434241231 +0000 UTC m=+22.754912818"
Sep 19 08:20:58.450053 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.450021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-56xxt" podStartSLOduration=7.891891146 podStartE2EDuration="22.450010644s" totalImagesPullingTime="14.558119498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:58.449499947 +0000 UTC m=+22.770171535" watchObservedRunningTime="2026-09-19 08:20:58.450010644 +0000 UTC m=+22.770682231"
Sep 19 08:20:58.733225 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:58.733202    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:20:59.114917 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.114722    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:20:58.733221654Z","UUID":"8f0b403f-a264-4d1f-9abd-9fb5348918ff","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:20:59.116359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.116334    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:20:59.116488 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.116369    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:20:59.363127 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.363090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj" event={"ID":"67a02ae7-ea30-4518-b05c-236b3630efe5","Type":"ContainerStarted","Data":"ef5f2e68eeaa581427810058c43a19a459c29821f56e4e916aa2d5f3351088d1"}
Sep 19 08:20:59.365177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.365122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cm2b2" event={"ID":"3921dcbc-5bec-4b64-a499-2a9340efaf3e","Type":"ContainerStarted","Data":"945b38e69d6417620cb7d4aa4aafeca930529ce7dcd7d3cee7f7c5cba1f36e1a"}
Sep 19 08:20:59.380711 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:20:59.380666    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cm2b2" podStartSLOduration=11.478975355 podStartE2EDuration="22.380648598s" totalImagesPullingTime="10.901673243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:59.380178911 +0000 UTC m=+23.700850500" watchObservedRunningTime="2026-09-19 08:20:59.380648598 +0000 UTC m=+23.701320189"
Sep 19 08:21:00.175733 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.175659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:00.175948 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.175662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:00.175948 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:00.175798    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:00.175948 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:00.175891    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:00.175948 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.175659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:00.176112 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:00.175994    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:00.371491 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.371454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"ee0ecc355fb920e49c11ff55364c635cb202922582799c951557549488cea773"}
Sep 19 08:21:00.373836 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.373801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj" event={"ID":"67a02ae7-ea30-4518-b05c-236b3630efe5","Type":"ContainerStarted","Data":"4136297491b901733bf983735c41d3bfb04d9d32954806e0fc9e872f64e63bf6"}
Sep 19 08:21:00.394961 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:00.394919    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sn5sj" podStartSLOduration=4.232747771 podStartE2EDuration="24.394900889s" totalImagesPullingTime="20.162153118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:00.394188862 +0000 UTC m=+24.714860450" watchObservedRunningTime="2026-09-19 08:21:00.394900889 +0000 UTC m=+24.715572477"
Sep 19 08:21:01.959703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:01.959665    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:21:01.960395 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:01.960373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:21:02.175570 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.175545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:02.175703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.175641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:02.175703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.175660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:02.175798 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:02.175643    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:02.175798 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:02.175755    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:02.175869 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:02.175846    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:02.383597 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.383225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" event={"ID":"d861b7f9-efdf-4245-b57b-9ac9973cca19","Type":"ContainerStarted","Data":"be0b25a6510007e5db99015a062671d6b73df2975708b1441e0eedbbe4cb834c"}
Sep 19 08:21:02.383597 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.383529    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:02.385804 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.385754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerStarted","Data":"fdb14c4c62cadd68ab81b1c7945be47f777a390d526fa78bc66dd5e22d722f86"}
Sep 19 08:21:02.386093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.386076    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:21:02.386495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.386474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-56xxt"
Sep 19 08:21:02.399762 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.399739    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:02.411664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:02.411609    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt" podStartSLOduration=7.296077481 podStartE2EDuration="25.41159107s" totalImagesPullingTime="18.115513589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:02.408720725 +0000 UTC m=+26.729392325" watchObservedRunningTime="2026-09-19 08:21:02.41159107 +0000 UTC m=+26.732262655"
Sep 19 08:21:03.390002 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:03.389968    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="fdb14c4c62cadd68ab81b1c7945be47f777a390d526fa78bc66dd5e22d722f86" exitCode=0
Sep 19 08:21:03.390520 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:03.390051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"fdb14c4c62cadd68ab81b1c7945be47f777a390d526fa78bc66dd5e22d722f86"}
Sep 19 08:21:03.391205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:03.390709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:03.391205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:03.390795    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:03.404932 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:03.404913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:04.119409 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.119382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dtq79"]
Sep 19 08:21:04.119530 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.119515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:04.119645 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:04.119623    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:04.123617 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.123100    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4vdw"]
Sep 19 08:21:04.123617 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.123235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:04.123617 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:04.123338    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:04.123848 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.123751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7m8x"]
Sep 19 08:21:04.123899 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.123851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:04.123953 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:04.123934    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:04.394944 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.394906    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="1688468eb1bd8b7d897f2ce7b9b079947c2347a88ae4337ca16893798a93cfce" exitCode=0
Sep 19 08:21:04.395541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:04.394974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"1688468eb1bd8b7d897f2ce7b9b079947c2347a88ae4337ca16893798a93cfce"}
Sep 19 08:21:05.399668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:05.399351    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="04f6ca9f3a8f86f71d5bc60e85c11fbe4f2ec5c8815d17b80a7bdd48f65d97e1" exitCode=0
Sep 19 08:21:05.399668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:05.399410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"04f6ca9f3a8f86f71d5bc60e85c11fbe4f2ec5c8815d17b80a7bdd48f65d97e1"}
Sep 19 08:21:06.176621 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:06.176594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:06.176734 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:06.176694    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:06.176860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:06.176842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:06.176908 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:06.176862    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:06.176970 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:06.176946    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:06.177048 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:06.177026    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:07.401354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:07.401315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:07.491374 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:07.491341    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:08.175683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.175575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:08.175683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.175619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:08.175683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.175575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:08.175993 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:08.175711    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:08.175993 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:08.175817    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:08.175993 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:08.175896    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:08.725338 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.725305    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:08.823541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.823446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:08.923846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:08.923813    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:09.779678 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:09.779611    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:09.811995 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:09.811969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:09.812107 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:09.812068    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:09.812146 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:09.812121    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:41.81210677 +0000 UTC m=+66.132778336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:09.879707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:09.879668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:09.912491 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:09.912463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:09.912632 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:09.912587    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:09.912690 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:09.912651    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret podName:16c58e2d-7fe3-436e-93d1-68bd2ed8385b nodeName:}" failed. No retries permitted until 2026-09-19 08:21:41.912635385 +0000 UTC m=+66.233306968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret") pod "global-pull-secret-syncer-h7m8x" (UID: "16c58e2d-7fe3-436e-93d1-68bd2ed8385b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:10.012973 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.012944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:10.013175 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.013064    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:10.013175 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.013083    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:10.013175 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.013095    2566 projected.go:196] Error preparing data for projected volume kube-api-access-vr7c4 for pod openshift-network-diagnostics/network-check-target-dtq79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:10.013175 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.013155    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4 podName:1563caa6-c545-42d5-a4e0-11abe8191e82 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.01313707 +0000 UTC m=+66.333808650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vr7c4" (UniqueName: "kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4") pod "network-check-target-dtq79" (UID: "1563caa6-c545-42d5-a4e0-11abe8191e82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:10.175687 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.175660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:10.175687 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.175672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:10.175687 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.175682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:10.175936 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.175761    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dtq79" podUID="1563caa6-c545-42d5-a4e0-11abe8191e82"
Sep 19 08:21:10.175936 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.175852    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7m8x" podUID="16c58e2d-7fe3-436e-93d1-68bd2ed8385b"
Sep 19 08:21:10.176002 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:10.175952    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:21:10.415058 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.414976    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="9db17ae3fb3446644082af0a8d66141a371ae04dc52f17caf3f4a502e15e8d5f" exitCode=0
Sep 19 08:21:10.415058 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.415030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"9db17ae3fb3446644082af0a8d66141a371ae04dc52f17caf3f4a502e15e8d5f"}
Sep 19 08:21:10.921276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.921251    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeReady"
Sep 19 08:21:10.921580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.921344    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:10.964871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.964818    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gfbdv"]
Sep 19 08:21:10.984942 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.984917    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8drv4"]
Sep 19 08:21:10.985076 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.985056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:10.987744 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.987721    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dzp9\""
Sep 19 08:21:10.987871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.987756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:10.987871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:10.987761    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:11.007053 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.007033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfbdv"]
Sep 19 08:21:11.007139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.007060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8drv4"]
Sep 19 08:21:11.007139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.007120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.009724 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.009708    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:11.009830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.009708    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:11.009957 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.009941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2s5wr\""
Sep 19 08:21:11.010007 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.009975    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:11.086808 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.086764    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.119758 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.119850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ssb\" (UniqueName: \"kubernetes.io/projected/57e04927-ba9e-458f-bd24-47f4bacc2896-kube-api-access-f5ssb\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.119850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tm66\" (UniqueName: \"kubernetes.io/projected/d47e6619-183e-4631-a052-55895ec8508f-kube-api-access-6tm66\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.119921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57e04927-ba9e-458f-bd24-47f4bacc2896-config-volume\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.119921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d47e6619-183e-4631-a052-55895ec8508f-cert\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.119921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.119911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e04927-ba9e-458f-bd24-47f4bacc2896-tmp-dir\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.188221 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.188194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.220576 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d47e6619-183e-4631-a052-55895ec8508f-cert\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.220576 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e04927-ba9e-458f-bd24-47f4bacc2896-tmp-dir\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.220677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.220714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ssb\" (UniqueName: \"kubernetes.io/projected/57e04927-ba9e-458f-bd24-47f4bacc2896-kube-api-access-f5ssb\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.220714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tm66\" (UniqueName: \"kubernetes.io/projected/d47e6619-183e-4631-a052-55895ec8508f-kube-api-access-6tm66\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.220815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.220727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57e04927-ba9e-458f-bd24-47f4bacc2896-config-volume\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.220864 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:11.220817    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.220903 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:11.220885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:11.720867176 +0000 UTC m=+36.041538767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.221070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.221054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e04927-ba9e-458f-bd24-47f4bacc2896-tmp-dir\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.221313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.221298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57e04927-ba9e-458f-bd24-47f4bacc2896-config-volume\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.224639 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.224624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d47e6619-183e-4631-a052-55895ec8508f-cert\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.233584 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.233553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ssb\" (UniqueName: \"kubernetes.io/projected/57e04927-ba9e-458f-bd24-47f4bacc2896-kube-api-access-f5ssb\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.233659 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.233619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tm66\" (UniqueName: \"kubernetes.io/projected/d47e6619-183e-4631-a052-55895ec8508f-kube-api-access-6tm66\") pod \"ingress-canary-8drv4\" (UID: \"d47e6619-183e-4631-a052-55895ec8508f\") " pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.307002 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.306980    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.314789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.314752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8drv4"
Sep 19 08:21:11.422186 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.422153    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d2b547b-9039-4c47-851b-9c6d065e56d8" containerID="057f18aa51aaa866afd91c121e8746df887a2d2c9d65ebce2f4a9cd3f3c9c9ad" exitCode=0
Sep 19 08:21:11.422292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.422204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerDied","Data":"057f18aa51aaa866afd91c121e8746df887a2d2c9d65ebce2f4a9cd3f3c9c9ad"}
Sep 19 08:21:11.441401 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.441370    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.638220 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.638196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8drv4"]
Sep 19 08:21:11.640912 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:21:11.640885    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47e6619_183e_4631_a052_55895ec8508f.slice/crio-eb4c52d6975b405633343877c8b87181376af666a9f24125ee2faaac3fefdfef WatchSource:0}: Error finding container eb4c52d6975b405633343877c8b87181376af666a9f24125ee2faaac3fefdfef: Status 404 returned error can't find the container with id eb4c52d6975b405633343877c8b87181376af666a9f24125ee2faaac3fefdfef
Sep 19 08:21:11.722511 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.722489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:11.722607 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:11.722594    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.722647 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:11.722642    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:12.72262544 +0000 UTC m=+37.043297009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.742401 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.742372    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.823256 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:11.823194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.175716 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.175640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:12.176561 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.175641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:12.176561 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.175641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:12.178553 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.178532    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:12.178675 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.178530    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:12.179593 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.179573    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:12.179707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.179592    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95gj6\""
Sep 19 08:21:12.179805 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.179750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:12.179890 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.179872    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5xtcl\""
Sep 19 08:21:12.428990 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.428918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" event={"ID":"7d2b547b-9039-4c47-851b-9c6d065e56d8","Type":"ContainerStarted","Data":"1ef07931b1954fa8f8b2b3c22b5e2ac32663261066fa4ca2af999e763adee26e"}
Sep 19 08:21:12.430223 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.430193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8drv4" event={"ID":"d47e6619-183e-4631-a052-55895ec8508f","Type":"ContainerStarted","Data":"eb4c52d6975b405633343877c8b87181376af666a9f24125ee2faaac3fefdfef"}
Sep 19 08:21:12.458932 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.458876    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8mlvx" podStartSLOduration=9.202227496 podStartE2EDuration="36.458858567s" totalImagesPullingTime="27.256631071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.457410886 +0000 UTC m=+36.778082475" watchObservedRunningTime="2026-09-19 08:21:12.458858567 +0000 UTC m=+36.779530155"
Sep 19 08:21:12.586369 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.586334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.685495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.685408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.728279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:12.728248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:12.728421 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:12.728375    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:12.728469 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:12.728435    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:14.728418589 +0000 UTC m=+39.049090167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:14.437236 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:14.436973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8drv4" event={"ID":"d47e6619-183e-4631-a052-55895ec8508f","Type":"ContainerStarted","Data":"ca778f347a8b32ba543ce249e4a65bbf075b9a78a5989d856148f3dd10078e61"}
Sep 19 08:21:14.453825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:14.453758    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8drv4" podStartSLOduration=2.070413992 podStartE2EDuration="4.453743912s" totalImagesPullingTime="2.38332992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:14.453187015 +0000 UTC m=+38.773858603" watchObservedRunningTime="2026-09-19 08:21:14.453743912 +0000 UTC m=+38.774415501"
Sep 19 08:21:14.745862 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:14.745791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:14.745981 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:14.745901    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:14.745981 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:14.745952    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.745939511 +0000 UTC m=+43.066611091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:15.127585 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:15.127507    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:16.192517 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:16.192478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:16.293908 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:16.293882    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:18.773004 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:18.772975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:18.773341 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:18.773090    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:18.773341 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:18.773140    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:26.773127003 +0000 UTC m=+51.093798569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:19.526913 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:19.526884    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:20.678426 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:20.678391    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:20.742583 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:20.742561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:24.863705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:24.863662    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:24.943289 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:24.943259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:25.659345 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:25.659312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:25.731353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:25.731322    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:26.823287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:26.823258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:26.823898 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:26.823405    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:26.823898 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:26.823491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.823470151 +0000 UTC m=+67.144141717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:34.214590 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:34.214548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:34.314156 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:34.314125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:35.409648 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:35.409619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clzwt"
Sep 19 08:21:35.569302 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:35.569270    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:35.634258 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:35.634231    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:41.825273 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:41.825238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:21:41.830259 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:41.827799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:41.835845 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:41.835829    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:21:41.835893 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:41.835885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:45.83586973 +0000 UTC m=+130.156541296 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : secret "metrics-daemon-secret" not found
Sep 19 08:21:41.925882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:41.925838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:41.928347 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:41.928328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:41.939531 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:41.939504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16c58e2d-7fe3-436e-93d1-68bd2ed8385b-original-pull-secret\") pod \"global-pull-secret-syncer-h7m8x\" (UID: \"16c58e2d-7fe3-436e-93d1-68bd2ed8385b\") " pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:42.026649 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.026624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:42.029158 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.029143    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:42.039032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.039014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:42.049968 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.049947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7c4\" (UniqueName: \"kubernetes.io/projected/1563caa6-c545-42d5-a4e0-11abe8191e82-kube-api-access-vr7c4\") pod \"network-check-target-dtq79\" (UID: \"1563caa6-c545-42d5-a4e0-11abe8191e82\") " pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:42.187260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.187240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7m8x"
Sep 19 08:21:42.196259 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.196245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5xtcl\""
Sep 19 08:21:42.204331 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.204313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:42.455951 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.455889    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7m8x"]
Sep 19 08:21:42.459150 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:21:42.459126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c58e2d_7fe3_436e_93d1_68bd2ed8385b.slice/crio-a25e049b04b02c5bfa0b16d7b0d95d71661636dcd983aa8119df06fa37c81966 WatchSource:0}: Error finding container a25e049b04b02c5bfa0b16d7b0d95d71661636dcd983aa8119df06fa37c81966: Status 404 returned error can't find the container with id a25e049b04b02c5bfa0b16d7b0d95d71661636dcd983aa8119df06fa37c81966
Sep 19 08:21:42.475509 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.475488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dtq79"]
Sep 19 08:21:42.477778 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:21:42.477745    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563caa6_c545_42d5_a4e0_11abe8191e82.slice/crio-65ab603292ed6bbebc3e08f0ecb7fe2104f8e150f3bed69e2716035d33e96028 WatchSource:0}: Error finding container 65ab603292ed6bbebc3e08f0ecb7fe2104f8e150f3bed69e2716035d33e96028: Status 404 returned error can't find the container with id 65ab603292ed6bbebc3e08f0ecb7fe2104f8e150f3bed69e2716035d33e96028
Sep 19 08:21:42.502207 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.502179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7m8x" event={"ID":"16c58e2d-7fe3-436e-93d1-68bd2ed8385b","Type":"ContainerStarted","Data":"a25e049b04b02c5bfa0b16d7b0d95d71661636dcd983aa8119df06fa37c81966"}
Sep 19 08:21:42.503245 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.503224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dtq79" event={"ID":"1563caa6-c545-42d5-a4e0-11abe8191e82","Type":"ContainerStarted","Data":"65ab603292ed6bbebc3e08f0ecb7fe2104f8e150f3bed69e2716035d33e96028"}
Sep 19 08:21:42.831131 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:42.831046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:21:42.831458 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:42.831231    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:42.831458 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:21:42.831292    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:14.831273417 +0000 UTC m=+99.151944989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:46.863870 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:46.863835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:46.955624 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:46.955542    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:47.517568 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:47.517537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dtq79" event={"ID":"1563caa6-c545-42d5-a4e0-11abe8191e82","Type":"ContainerStarted","Data":"f09bf9d7a1be04e7414066572668125ff794ad91d53984a120d676082ec11c87"}
Sep 19 08:21:47.517827 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:47.517640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:21:47.519104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:47.519081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7m8x" event={"ID":"16c58e2d-7fe3-436e-93d1-68bd2ed8385b","Type":"ContainerStarted","Data":"7264bb7e859e443f5f5ddcb02e246e418e9ea3e4e4297b7e81a2722074e44074"}
Sep 19 08:21:47.540655 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:47.540615    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dtq79" podStartSLOduration=66.337049178 podStartE2EDuration="1m10.540604103s" totalImagesPullingTime="4.203554925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:47.539220135 +0000 UTC m=+71.859891722" watchObservedRunningTime="2026-09-19 08:21:47.540604103 +0000 UTC m=+71.861275690"
Sep 19 08:21:47.557036 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:21:47.556996    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h7m8x" podStartSLOduration=66.330276036 podStartE2EDuration="1m10.556983812s" totalImagesPullingTime="4.226707776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:47.556747837 +0000 UTC m=+71.877419425" watchObservedRunningTime="2026-09-19 08:21:47.556983812 +0000 UTC m=+71.877655378"
Sep 19 08:22:04.408594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.408481    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"]
Sep 19 08:22:04.412391 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.412370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.417259 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.417238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 08:22:04.417380 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.417262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nnjtg\""
Sep 19 08:22:04.417380 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.417276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 08:22:04.417380 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.417307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 19 08:22:04.417380 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.417346    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 08:22:04.422845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.422825    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"]
Sep 19 08:22:04.448450 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.448427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"]
Sep 19 08:22:04.451065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.451051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.462072 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.462056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 08:22:04.463122 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.463106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdfcm\" (UniqueName: \"kubernetes.io/projected/49643217-fd69-47c8-bc6f-144da32df3a1-kube-api-access-wdfcm\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.463185 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.463139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49643217-fd69-47c8-bc6f-144da32df3a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.473451 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.473430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"]
Sep 19 08:22:04.563481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.563444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-klusterlet-config\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.564125 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.563502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdfcm\" (UniqueName: \"kubernetes.io/projected/49643217-fd69-47c8-bc6f-144da32df3a1-kube-api-access-wdfcm\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.564125 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.563554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49643217-fd69-47c8-bc6f-144da32df3a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.564125 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.563585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65z7\" (UniqueName: \"kubernetes.io/projected/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-kube-api-access-q65z7\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.564125 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.563615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-tmp\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.566112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.566084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49643217-fd69-47c8-bc6f-144da32df3a1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.574831 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.574808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdfcm\" (UniqueName: \"kubernetes.io/projected/49643217-fd69-47c8-bc6f-144da32df3a1-kube-api-access-wdfcm\") pod \"managed-serviceaccount-addon-agent-f444848fd-zk4p2\" (UID: \"49643217-fd69-47c8-bc6f-144da32df3a1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.663852 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.663796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65z7\" (UniqueName: \"kubernetes.io/projected/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-kube-api-access-q65z7\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.663852 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.663832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-tmp\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.663968 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.663857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-klusterlet-config\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.664226 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.664201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-tmp\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.666021 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.666002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-klusterlet-config\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.671560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.671542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65z7\" (UniqueName: \"kubernetes.io/projected/6ca64971-1f95-4a28-83b9-c6ebc0ac8388-kube-api-access-q65z7\") pod \"klusterlet-addon-workmgr-6759bf775-6ldf5\" (UID: \"6ca64971-1f95-4a28-83b9-c6ebc0ac8388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:04.743791 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.743758    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"
Sep 19 08:22:04.758844 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:04.758825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:05.017282 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:05.017172    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2"]
Sep 19 08:22:05.021472 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:22:05.021447    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49643217_fd69_47c8_bc6f_144da32df3a1.slice/crio-6d9f678df74036ecc34137da508d95422466f9cb404310ee1829fc6a4e75273c WatchSource:0}: Error finding container 6d9f678df74036ecc34137da508d95422466f9cb404310ee1829fc6a4e75273c: Status 404 returned error can't find the container with id 6d9f678df74036ecc34137da508d95422466f9cb404310ee1829fc6a4e75273c
Sep 19 08:22:05.039552 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:05.039527    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"]
Sep 19 08:22:05.042870 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:22:05.042846    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca64971_1f95_4a28_83b9_c6ebc0ac8388.slice/crio-06250781e38b15b1c2f64eb69775198855ee25ec37ee6a610e7ee7b15cc24fc8 WatchSource:0}: Error finding container 06250781e38b15b1c2f64eb69775198855ee25ec37ee6a610e7ee7b15cc24fc8: Status 404 returned error can't find the container with id 06250781e38b15b1c2f64eb69775198855ee25ec37ee6a610e7ee7b15cc24fc8
Sep 19 08:22:05.562778 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:05.562736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2" event={"ID":"49643217-fd69-47c8-bc6f-144da32df3a1","Type":"ContainerStarted","Data":"6d9f678df74036ecc34137da508d95422466f9cb404310ee1829fc6a4e75273c"}
Sep 19 08:22:05.563819 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:05.563801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" event={"ID":"6ca64971-1f95-4a28-83b9-c6ebc0ac8388","Type":"ContainerStarted","Data":"06250781e38b15b1c2f64eb69775198855ee25ec37ee6a610e7ee7b15cc24fc8"}
Sep 19 08:22:09.576828 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.576789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2" event={"ID":"49643217-fd69-47c8-bc6f-144da32df3a1","Type":"ContainerStarted","Data":"888a8e1ad24344c1d6e2c9569d6f58b66401e3f71c142475a1702966506df8ce"}
Sep 19 08:22:09.578198 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.578173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" event={"ID":"6ca64971-1f95-4a28-83b9-c6ebc0ac8388","Type":"ContainerStarted","Data":"70c9cd2057561df2eeb6c1386cdd17c71347992c9676fe753ae7593094c1ea92"}
Sep 19 08:22:09.578409 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.578389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:09.579603 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.579585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:22:09.593003 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.592962    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2" podStartSLOduration=1.194059665 podStartE2EDuration="5.592951336s" totalImagesPullingTime="4.398891671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:09.59203555 +0000 UTC m=+93.912707136" watchObservedRunningTime="2026-09-19 08:22:09.592951336 +0000 UTC m=+93.913622922"
Sep 19 08:22:09.610438 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:09.610403    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" podStartSLOduration=1.221583151 podStartE2EDuration="5.61039173s" totalImagesPullingTime="4.388808579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:09.609945857 +0000 UTC m=+93.930617436" watchObservedRunningTime="2026-09-19 08:22:09.61039173 +0000 UTC m=+93.931063316"
Sep 19 08:22:14.835912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:14.835879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:22:14.836249 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:22:14.836014    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:22:14.836249 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:22:14.836071    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:23:18.836056655 +0000 UTC m=+163.156728220 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:22:18.524565 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:18.524530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dtq79"
Sep 19 08:22:25.143260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:25.143223    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.217785 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:25.217735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:45.840095 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:22:45.840059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:22:45.840464 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:22:45.840179    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:22:45.840464 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:22:45.840244    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:24:47.840227458 +0000 UTC m=+252.160899025 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : secret "metrics-daemon-secret" not found
Sep 19 08:23:13.993717 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:23:13.993612    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gfbdv" podUID="57e04927-ba9e-458f-bd24-47f4bacc2896"
Sep 19 08:23:14.735666 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:23:14.735635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:23:15.199876 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:23:15.199830    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:23:18.860114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:23:18.860063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:23:18.860519 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:23:18.860220    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:23:18.860519 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:23:18.860297    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls podName:57e04927-ba9e-458f-bd24-47f4bacc2896 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:20.86027725 +0000 UTC m=+285.180948818 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls") pod "dns-default-gfbdv" (UID: "57e04927-ba9e-458f-bd24-47f4bacc2896") : secret "dns-default-metrics-tls" not found
Sep 19 08:23:30.176211 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:23:30.176125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:24:09.579218 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.579184    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" podUID="6ca64971-1f95-4a28-83b9-c6ebc0ac8388" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 19 08:24:09.867049 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.867022    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ca64971-1f95-4a28-83b9-c6ebc0ac8388" containerID="70c9cd2057561df2eeb6c1386cdd17c71347992c9676fe753ae7593094c1ea92" exitCode=1
Sep 19 08:24:09.867202 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.867100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" event={"ID":"6ca64971-1f95-4a28-83b9-c6ebc0ac8388","Type":"ContainerDied","Data":"70c9cd2057561df2eeb6c1386cdd17c71347992c9676fe753ae7593094c1ea92"}
Sep 19 08:24:09.867419 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.867402    2566 scope.go:117] "RemoveContainer" containerID="70c9cd2057561df2eeb6c1386cdd17c71347992c9676fe753ae7593094c1ea92"
Sep 19 08:24:09.868649 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.868533    2566 generic.go:356] "Generic (PLEG): container finished" podID="49643217-fd69-47c8-bc6f-144da32df3a1" containerID="888a8e1ad24344c1d6e2c9569d6f58b66401e3f71c142475a1702966506df8ce" exitCode=255
Sep 19 08:24:09.868649 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.868564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2" event={"ID":"49643217-fd69-47c8-bc6f-144da32df3a1","Type":"ContainerDied","Data":"888a8e1ad24344c1d6e2c9569d6f58b66401e3f71c142475a1702966506df8ce"}
Sep 19 08:24:09.874711 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:09.874696    2566 scope.go:117] "RemoveContainer" containerID="888a8e1ad24344c1d6e2c9569d6f58b66401e3f71c142475a1702966506df8ce"
Sep 19 08:24:10.874143 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:10.874101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5" event={"ID":"6ca64971-1f95-4a28-83b9-c6ebc0ac8388","Type":"ContainerStarted","Data":"7815af96668d4e8d3c07b1c23bc062f39d7c0da2ffe06f46886024ddcb2bd186"}
Sep 19 08:24:10.874599 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:10.874385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:24:10.875040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:10.875013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6759bf775-6ldf5"
Sep 19 08:24:10.875953 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:10.875932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f444848fd-zk4p2" event={"ID":"49643217-fd69-47c8-bc6f-144da32df3a1","Type":"ContainerStarted","Data":"1a27716b9f9aae7ca415b1be9e7cb13b29a550727fb5708bedf5c4699b68f158"}
Sep 19 08:24:47.852493 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:24:47.852449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:24:47.852986 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:24:47.852563    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:24:47.852986 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:24:47.852615    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs podName:f6df42b6-9d4c-4388-8750-2b7e19ccbe71 nodeName:}" failed. No retries permitted until 2026-09-19 08:26:49.852601668 +0000 UTC m=+374.173273248 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs") pod "network-metrics-daemon-c4vdw" (UID: "f6df42b6-9d4c-4388-8750-2b7e19ccbe71") : secret "metrics-daemon-secret" not found
Sep 19 08:25:03.458330 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.458251    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:03.461836 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.461821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.464490 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.464471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 08:25:03.464808 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.464790    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 08:25:03.465119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.465099    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 08:25:03.465198 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.465160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lv4hf\""
Sep 19 08:25:03.470121 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.470103    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 08:25:03.487118 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.487097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:03.546495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.546465    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"]
Sep 19 08:25:03.549255 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.549238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.552576 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.552559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.552741 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.552718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.553158 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.553136    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 19 08:25:03.553234 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.553223    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 19 08:25:03.553278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.553258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8c8vp\""
Sep 19 08:25:03.559297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9skx\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.559595 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.559469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.563827 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.563808    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"]
Sep 19 08:25:03.660439 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cdd2703-c202-41de-acf8-46444e1feda6-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2r9\" (UniqueName: \"kubernetes.io/projected/1cdd2703-c202-41de-acf8-46444e1feda6-kube-api-access-6r2r9\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:03.660576    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660623 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:03.660597    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9d78fcf6-lm4wd: secret "image-registry-tls" not found
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:03.660657    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls podName:4649de07-9935-42d6-bb85-e44884a90b14 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:04.160637422 +0000 UTC m=+268.481308992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls") pod "image-registry-b9d78fcf6-lm4wd" (UID: "4649de07-9935-42d6-bb85-e44884a90b14") : secret "image-registry-tls" not found
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9skx\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.660981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.660943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cdd2703-c202-41de-acf8-46444e1feda6-config\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.661221 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.661202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.661287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.661250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.661732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.661712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.663348 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.663324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.663429 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.663324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.669934 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.669914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.670158 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.670142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9skx\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:03.723183 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.723110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:03.761524 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.761501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cdd2703-c202-41de-acf8-46444e1feda6-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.761594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.761527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2r9\" (UniqueName: \"kubernetes.io/projected/1cdd2703-c202-41de-acf8-46444e1feda6-kube-api-access-6r2r9\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.761594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.761558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cdd2703-c202-41de-acf8-46444e1feda6-config\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.762002 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.761987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cdd2703-c202-41de-acf8-46444e1feda6-config\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.763560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.763534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cdd2703-c202-41de-acf8-46444e1feda6-serving-cert\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.769381 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.769362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2r9\" (UniqueName: \"kubernetes.io/projected/1cdd2703-c202-41de-acf8-46444e1feda6-kube-api-access-6r2r9\") pod \"service-ca-operator-57cdc4b768-rnzpp\" (UID: \"1cdd2703-c202-41de-acf8-46444e1feda6\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:03.783113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.783088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:03.858451 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:03.858428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"
Sep 19 08:25:04.127266 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:04.127218    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp"]
Sep 19 08:25:04.131745 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:04.131724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdd2703_c202_41de_acf8_46444e1feda6.slice/crio-ac2a28e53e140d7926c82bcd804dd06275d4679dc3d1e8b3685c8347f1e6766b WatchSource:0}: Error finding container ac2a28e53e140d7926c82bcd804dd06275d4679dc3d1e8b3685c8347f1e6766b: Status 404 returned error can't find the container with id ac2a28e53e140d7926c82bcd804dd06275d4679dc3d1e8b3685c8347f1e6766b
Sep 19 08:25:04.164310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:04.164283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:04.164398 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:04.164387    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:04.164444 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:04.164400    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9d78fcf6-lm4wd: secret "image-registry-tls" not found
Sep 19 08:25:04.164444 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:04.164442    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls podName:4649de07-9935-42d6-bb85-e44884a90b14 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:05.164428386 +0000 UTC m=+269.485099957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls") pod "image-registry-b9d78fcf6-lm4wd" (UID: "4649de07-9935-42d6-bb85-e44884a90b14") : secret "image-registry-tls" not found
Sep 19 08:25:05.030209 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:05.030175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp" event={"ID":"1cdd2703-c202-41de-acf8-46444e1feda6","Type":"ContainerStarted","Data":"ac2a28e53e140d7926c82bcd804dd06275d4679dc3d1e8b3685c8347f1e6766b"}
Sep 19 08:25:05.172120 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:05.172082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:05.172288 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:05.172223    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:05.172288 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:05.172240    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9d78fcf6-lm4wd: secret "image-registry-tls" not found
Sep 19 08:25:05.172402 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:05.172297    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls podName:4649de07-9935-42d6-bb85-e44884a90b14 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:07.172283297 +0000 UTC m=+271.492954866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls") pod "image-registry-b9d78fcf6-lm4wd" (UID: "4649de07-9935-42d6-bb85-e44884a90b14") : secret "image-registry-tls" not found
Sep 19 08:25:05.415378 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:05.415345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:05.517456 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:05.517413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:07.037987 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.037956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp" event={"ID":"1cdd2703-c202-41de-acf8-46444e1feda6","Type":"ContainerStarted","Data":"1db283e7a36605b15389c9302937acefcf7146ec50679e419c6bb6898e43e79a"}
Sep 19 08:25:07.064714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.064663    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp" podStartSLOduration=2.036971816 podStartE2EDuration="4.064649541s" totalImagesPullingTime="2.027677725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:07.063592721 +0000 UTC m=+271.384264308" watchObservedRunningTime="2026-09-19 08:25:07.064649541 +0000 UTC m=+271.385321129"
Sep 19 08:25:07.183164 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.183134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:07.183351 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:07.183329    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:07.183393 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:07.183355    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9d78fcf6-lm4wd: secret "image-registry-tls" not found
Sep 19 08:25:07.183429 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:07.183420    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls podName:4649de07-9935-42d6-bb85-e44884a90b14 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:11.183400947 +0000 UTC m=+275.504072520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls") pod "image-registry-b9d78fcf6-lm4wd" (UID: "4649de07-9935-42d6-bb85-e44884a90b14") : secret "image-registry-tls" not found
Sep 19 08:25:07.470572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.470536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:07.529113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.529087    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:07.939900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:07.939867    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.041600 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.041566    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.101346 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.101325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.178447 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.178419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.279788 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.279694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.788870 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.788842    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4gxjl"]
Sep 19 08:25:08.791601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.791581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.794546 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.794524    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:25:08.794546 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.794552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:25:08.795562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.795545    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:25:08.795562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.795561    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mq7r\""
Sep 19 08:25:08.795721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.795604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:25:08.803918 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.803898    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4gxjl"]
Sep 19 08:25:08.814690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.814671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.882071 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.882044    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:08.896167 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.896147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.896230 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.896181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c6f8397-5311-42a8-84b1-6416370ef901-data-volume\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.896230 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.896211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c6f8397-5311-42a8-84b1-6416370ef901-crio-socket\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.896291 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.896265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zfs9\" (UniqueName: \"kubernetes.io/projected/4c6f8397-5311-42a8-84b1-6416370ef901-kube-api-access-4zfs9\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.896291 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.896283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996674 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zfs9\" (UniqueName: \"kubernetes.io/projected/4c6f8397-5311-42a8-84b1-6416370ef901-kube-api-access-4zfs9\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c6f8397-5311-42a8-84b1-6416370ef901-data-volume\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c6f8397-5311-42a8-84b1-6416370ef901-crio-socket\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:08.996834    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:08.996860    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:08.996904    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:09.496884589 +0000 UTC m=+273.817556169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.996905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c6f8397-5311-42a8-84b1-6416370ef901-crio-socket\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:08.996966 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:08.996923    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:09.496913472 +0000 UTC m=+273.817585041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:08.997199 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:08.997085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c6f8397-5311-42a8-84b1-6416370ef901-data-volume\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:09.006924 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.006899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zfs9\" (UniqueName: \"kubernetes.io/projected/4c6f8397-5311-42a8-84b1-6416370ef901-kube-api-access-4zfs9\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:09.061465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.061400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:09.116894 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.116869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:09.500723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.500684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:09.500867 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.500789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:09.500867 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:09.500840    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.500990 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:09.500899    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:10.500885526 +0000 UTC m=+274.821557092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.500990 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:09.500902    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.500990 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:09.500958    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:10.500940208 +0000 UTC m=+274.821611792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.805333 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.805251    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:09.860786 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:09.860738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:10.509589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:10.509552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:10.510036 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:10.509614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:10.510036 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:10.509669    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:10.510036 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:10.509712    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:12.509700116 +0000 UTC m=+276.830371681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:10.510036 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:10.509718    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:10.510036 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:10.509818    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:12.509801719 +0000 UTC m=+276.830473288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:10.739764 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:10.739730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:10.958739 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:10.958711    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vf9n9_f1704028-5d63-44ab-b038-4867a45a8896/dns-node-resolver/0.log"
Sep 19 08:25:11.215826 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:11.215747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:11.215948 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:11.215878    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:11.215948 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:11.215896    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9d78fcf6-lm4wd: secret "image-registry-tls" not found
Sep 19 08:25:11.215948 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:11.215946    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls podName:4649de07-9935-42d6-bb85-e44884a90b14 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:19.215932154 +0000 UTC m=+283.536603723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls") pod "image-registry-b9d78fcf6-lm4wd" (UID: "4649de07-9935-42d6-bb85-e44884a90b14") : secret "image-registry-tls" not found
Sep 19 08:25:11.560638 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:11.560570    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hdz7_e93a1df2-a189-4c33-aad1-d17ca2d7cc1d/node-ca/0.log"
Sep 19 08:25:12.526480 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:12.526450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:12.526641 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:12.526506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:12.526641 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:12.526588    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:12.526641 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:12.526599    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:12.526641 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:12.526638    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:16.526624875 +0000 UTC m=+280.847296454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:12.526798 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:12.526684    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:16.526671449 +0000 UTC m=+280.847343020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:12.559935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:12.559911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8drv4_d47e6619-183e-4631-a052-55895ec8508f/serve-healthcheck-canary/0.log"
Sep 19 08:25:13.297450 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:13.297415    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:14.427310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:14.427279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:14.669882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:14.669850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:14.867576 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:14.867536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:14.968758 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:14.968728    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:15.071249 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:15.071222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:15.170764 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:15.170681    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:15.916111 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:15.916083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:16.114617 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.114564    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:16.517004 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.516975    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:16.557190 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.557152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:16.557315 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.557217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:16.557315 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:16.557267    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:16.557315 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:16.557301    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:16.557415 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:16.557321    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:24.557307663 +0000 UTC m=+288.877979229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:16.557415 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:16.557343    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls podName:4c6f8397-5311-42a8-84b1-6416370ef901 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:24.557331311 +0000 UTC m=+288.878002880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4gxjl" (UID: "4c6f8397-5311-42a8-84b1-6416370ef901") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:16.917109 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.917083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:16.971185 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:16.971163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:17.717334 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:17.717300    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:17.736138 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:17.736113    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gfbdv" podUID="57e04927-ba9e-458f-bd24-47f4bacc2896"
Sep 19 08:25:17.914831 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:17.914805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:18.068425 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:18.068367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:18.229917 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:18.229890    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:18.330271 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:18.330204    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:18.382779 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:18.382751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:18.519219 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:18.519186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.128831 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.128800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.277983 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.277950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:19.280598 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.280565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"image-registry-b9d78fcf6-lm4wd\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") " pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:19.370565 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.370542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:19.644999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.644940    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:19.647262 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:19.647240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4649de07_9935_42d6_bb85_e44884a90b14.slice/crio-ffaa365310202c0804ca127a9810642539b09214b0f7df3e9b837059b1302ba2 WatchSource:0}: Error finding container ffaa365310202c0804ca127a9810642539b09214b0f7df3e9b837059b1302ba2: Status 404 returned error can't find the container with id ffaa365310202c0804ca127a9810642539b09214b0f7df3e9b837059b1302ba2
Sep 19 08:25:19.715680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.715655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.926382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:19.926308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:20.081754 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.081720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" event={"ID":"4649de07-9935-42d6-bb85-e44884a90b14","Type":"ContainerStarted","Data":"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"}
Sep 19 08:25:20.081754 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.081782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" event={"ID":"4649de07-9935-42d6-bb85-e44884a90b14","Type":"ContainerStarted","Data":"ffaa365310202c0804ca127a9810642539b09214b0f7df3e9b837059b1302ba2"}
Sep 19 08:25:20.081969 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.081944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:20.100346 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.100297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" podStartSLOduration=17.100283802 podStartE2EDuration="17.100283802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:20.099438171 +0000 UTC m=+284.420109759" watchObservedRunningTime="2026-09-19 08:25:20.100283802 +0000 UTC m=+284.420955390"
Sep 19 08:25:20.319447 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.319363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:20.510182 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.510150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:20.892697 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.892662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:20.895044 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.895017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57e04927-ba9e-458f-bd24-47f4bacc2896-metrics-tls\") pod \"dns-default-gfbdv\" (UID: \"57e04927-ba9e-458f-bd24-47f4bacc2896\") " pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:20.901803 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.901764    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:20.965248 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:20.965212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:21.071377 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:21.071356    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dzp9\""
Sep 19 08:25:21.079267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:21.079247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:21.133463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:21.133432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:21.353073 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:21.353014    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfbdv"]
Sep 19 08:25:21.355655 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:21.355631    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e04927_ba9e_458f_bd24_47f4bacc2896.slice/crio-6068080e779dda0174a3d6e938ab6e10cf8d192b02688451f82c5f4ff30cdc7b WatchSource:0}: Error finding container 6068080e779dda0174a3d6e938ab6e10cf8d192b02688451f82c5f4ff30cdc7b: Status 404 returned error can't find the container with id 6068080e779dda0174a3d6e938ab6e10cf8d192b02688451f82c5f4ff30cdc7b
Sep 19 08:25:21.516403 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:21.516374    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:22.089348 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:22.089318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfbdv" event={"ID":"57e04927-ba9e-458f-bd24-47f4bacc2896","Type":"ContainerStarted","Data":"6068080e779dda0174a3d6e938ab6e10cf8d192b02688451f82c5f4ff30cdc7b"}
Sep 19 08:25:22.110803 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:22.110752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:22.309694 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:22.309658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:22.716624 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:22.716590    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:22.919463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:22.919424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:23.094690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:23.094670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfbdv" event={"ID":"57e04927-ba9e-458f-bd24-47f4bacc2896","Type":"ContainerStarted","Data":"0f6b909ddb42b6aa869742c68c8958ad8cb83d08ea67bd647a68f5fe0bc3394e"}
Sep 19 08:25:23.316755 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:23.316683    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:23.932028 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:23.931985    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:24.100014 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.099983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfbdv" event={"ID":"57e04927-ba9e-458f-bd24-47f4bacc2896","Type":"ContainerStarted","Data":"35110227cd9dc1ddcf6dbbf6d7bdfb1ea8d601923f678a082ed4e3016fa582eb"}
Sep 19 08:25:24.100184 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.100110    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:24.119059 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.119011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gfbdv" podStartSLOduration=252.470441317 podStartE2EDuration="4m14.118996167s" totalImagesPullingTime="1.64855485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:24.118338108 +0000 UTC m=+288.439009697" watchObservedRunningTime="2026-09-19 08:25:24.118996167 +0000 UTC m=+288.439667756"
Sep 19 08:25:24.316714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.316633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:24.620830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.620801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:24.620996 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.620849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:24.621332 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.621315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c6f8397-5311-42a8-84b1-6416370ef901-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:24.623068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.623044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c6f8397-5311-42a8-84b1-6416370ef901-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4gxjl\" (UID: \"4c6f8397-5311-42a8-84b1-6416370ef901\") " pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:24.700615 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.700587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4gxjl"
Sep 19 08:25:24.714634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.714606    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:24.922845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.922739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:24.971828 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:24.971800    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4gxjl"]
Sep 19 08:25:24.974334 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:24.974307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6f8397_5311_42a8_84b1_6416370ef901.slice/crio-f54f82fcb7b3d1a30448629184ebcb103240f6a4acecaa4351c47e1ed981b2ee WatchSource:0}: Error finding container f54f82fcb7b3d1a30448629184ebcb103240f6a4acecaa4351c47e1ed981b2ee: Status 404 returned error can't find the container with id f54f82fcb7b3d1a30448629184ebcb103240f6a4acecaa4351c47e1ed981b2ee
Sep 19 08:25:25.104727 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:25.104687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gxjl" event={"ID":"4c6f8397-5311-42a8-84b1-6416370ef901","Type":"ContainerStarted","Data":"ddb03af53fcb51c249ac82c57040ef9539ece7f54b57b1e5ea3d8f52cb4cb368"}
Sep 19 08:25:25.104891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:25.104743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gxjl" event={"ID":"4c6f8397-5311-42a8-84b1-6416370ef901","Type":"ContainerStarted","Data":"f54f82fcb7b3d1a30448629184ebcb103240f6a4acecaa4351c47e1ed981b2ee"}
Sep 19 08:25:25.112951 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:25.112919    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:26.109504 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:26.109416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gxjl" event={"ID":"4c6f8397-5311-42a8-84b1-6416370ef901","Type":"ContainerStarted","Data":"98288f231b31a3c3a2c88265fec9c560a17255ece699ae0dd943b35dd345100d"}
Sep 19 08:25:27.114533 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:27.114506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4gxjl" event={"ID":"4c6f8397-5311-42a8-84b1-6416370ef901","Type":"ContainerStarted","Data":"df73b90efdf0055a1e275ee3a68747f8a7b2afc210b09bddaa5ceac94d996366"}
Sep 19 08:25:27.130881 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:27.130716    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4gxjl" podStartSLOduration=17.190093706 podStartE2EDuration="19.130699017s" totalImagesPullingTime="1.940605311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:25:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:27.130483208 +0000 UTC m=+291.451154796" watchObservedRunningTime="2026-09-19 08:25:27.130699017 +0000 UTC m=+291.451370606"
Sep 19 08:25:30.819713 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.819677    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:30.855179 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.855149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c5868599f-tqmnc"]
Sep 19 08:25:30.858070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.858055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.870276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.870254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5868599f-tqmnc"]
Sep 19 08:25:30.888838 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.888816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:30.964765 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-tls\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.964856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-certificates\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.964856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbss\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-kube-api-access-4cbss\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.964937 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-bound-sa-token\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.964937 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-trusted-ca\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.965015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-ca-trust-extracted\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.965015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-image-registry-private-configuration\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.965015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.964986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-installation-pull-secrets\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:30.994655 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:30.994632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:31.066000 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.065972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbss\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-kube-api-access-4cbss\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066124 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-bound-sa-token\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066124 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-trusted-ca\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066124 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-ca-trust-extracted\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066124 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-image-registry-private-configuration\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066338 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-installation-pull-secrets\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066338 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-tls\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066338 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-certificates\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.066560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-ca-trust-extracted\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.067019 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.066995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-certificates\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.067193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.067173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-trusted-ca\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.068644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.068625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-registry-tls\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.069064 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.069044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-image-registry-private-configuration\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.069224 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.069202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-installation-pull-secrets\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.073560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.073517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-bound-sa-token\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.073911 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.073886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbss\" (UniqueName: \"kubernetes.io/projected/c6a8e307-6e55-42ed-b5d0-8b0067caf21a-kube-api-access-4cbss\") pod \"image-registry-5c5868599f-tqmnc\" (UID: \"c6a8e307-6e55-42ed-b5d0-8b0067caf21a\") " pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.166946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.166917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:31.186444 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.186417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:31.286310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.286280    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:31.436273 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:31.436250    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5868599f-tqmnc"]
Sep 19 08:25:31.438609 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:31.438584    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a8e307_6e55_42ed_b5d0_8b0067caf21a.slice/crio-ce88c2f17ad577d0e5e343435b1824894b42d9626deefebf2d96425b580ee208 WatchSource:0}: Error finding container ce88c2f17ad577d0e5e343435b1824894b42d9626deefebf2d96425b580ee208: Status 404 returned error can't find the container with id ce88c2f17ad577d0e5e343435b1824894b42d9626deefebf2d96425b580ee208
Sep 19 08:25:32.131354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:32.131320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc" event={"ID":"c6a8e307-6e55-42ed-b5d0-8b0067caf21a","Type":"ContainerStarted","Data":"bc6d53e65c832996f1f8f6d8a6b6ec37598efe1686f974e84ff07412f8d142dd"}
Sep 19 08:25:32.131757 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:32.131368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc" event={"ID":"c6a8e307-6e55-42ed-b5d0-8b0067caf21a","Type":"ContainerStarted","Data":"ce88c2f17ad577d0e5e343435b1824894b42d9626deefebf2d96425b580ee208"}
Sep 19 08:25:32.131757 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:32.131447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:32.150303 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:32.150260    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc" podStartSLOduration=2.150245326 podStartE2EDuration="2.150245326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:32.14940687 +0000 UTC m=+296.470078456" watchObservedRunningTime="2026-09-19 08:25:32.150245326 +0000 UTC m=+296.470916914"
Sep 19 08:25:33.176782 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:33.176743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c4vdw" podUID="f6df42b6-9d4c-4388-8750-2b7e19ccbe71"
Sep 19 08:25:34.107276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:34.107245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gfbdv"
Sep 19 08:25:36.048123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:36.048095    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:25:37.656382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:37.656344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:38.888425 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:38.888385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:40.826302 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.826261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:40.826824 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.826800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:40.864104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.864076    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"]
Sep 19 08:25:40.867859 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.867841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:40.870556 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8xcdl\""
Sep 19 08:25:40.870676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:25:40.870676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870619    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 19 08:25:40.870802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870688    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 19 08:25:40.870802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:25:40.870915 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.870880    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:25:40.876107 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.876087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"]
Sep 19 08:25:40.895114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.895086    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:40.933189 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.933152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmjm\" (UniqueName: \"kubernetes.io/projected/d60b80c4-7785-49b5-9c57-6539dbcc0d95-kube-api-access-xkmjm\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:40.933323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.933207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:40.933323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.933253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:40.933323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:40.933273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60b80c4-7785-49b5-9c57-6539dbcc0d95-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.034588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.034557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60b80c4-7785-49b5-9c57-6539dbcc0d95-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.034749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.034600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmjm\" (UniqueName: \"kubernetes.io/projected/d60b80c4-7785-49b5-9c57-6539dbcc0d95-kube-api-access-xkmjm\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.034749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.034641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.034749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.034667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.034903 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:41.034800    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 19 08:25:41.034903 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:41.034881    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls podName:d60b80c4-7785-49b5-9c57-6539dbcc0d95 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:41.534861712 +0000 UTC m=+305.855533289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-c2jqk" (UID: "d60b80c4-7785-49b5-9c57-6539dbcc0d95") : secret "prometheus-operator-tls" not found
Sep 19 08:25:41.035215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.035188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60b80c4-7785-49b5-9c57-6539dbcc0d95-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.037048 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.037020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.044218 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.044197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmjm\" (UniqueName: \"kubernetes.io/projected/d60b80c4-7785-49b5-9c57-6539dbcc0d95-kube-api-access-xkmjm\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.175203 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.175173    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:41.538390 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.538292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.540749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.540724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d60b80c4-7785-49b5-9c57-6539dbcc0d95-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c2jqk\" (UID: \"d60b80c4-7785-49b5-9c57-6539dbcc0d95\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:41.777853 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:41.777818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"
Sep 19 08:25:42.049360 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:42.049284    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk"]
Sep 19 08:25:42.052006 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:42.051977    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60b80c4_7785_49b5_9c57_6539dbcc0d95.slice/crio-9de8b24e3ecd84d5529c29c1d600c4e2243b496b0f52a8b3742436de59f3f1da WatchSource:0}: Error finding container 9de8b24e3ecd84d5529c29c1d600c4e2243b496b0f52a8b3742436de59f3f1da: Status 404 returned error can't find the container with id 9de8b24e3ecd84d5529c29c1d600c4e2243b496b0f52a8b3742436de59f3f1da
Sep 19 08:25:42.054078 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:42.054061    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:25:42.162117 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:42.162079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk" event={"ID":"d60b80c4-7785-49b5-9c57-6539dbcc0d95","Type":"ContainerStarted","Data":"9de8b24e3ecd84d5529c29c1d600c4e2243b496b0f52a8b3742436de59f3f1da"}
Sep 19 08:25:43.233160 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:43.233123    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:43.288833 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:43.288800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:43.588108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:43.588021    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:44.171069 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:44.171026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk" event={"ID":"d60b80c4-7785-49b5-9c57-6539dbcc0d95","Type":"ContainerStarted","Data":"7ed8845aa41c859752800c58ad9e546ac86d8825bbd055bc404360256eda8485"}
Sep 19 08:25:44.171212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:44.171131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk" event={"ID":"d60b80c4-7785-49b5-9c57-6539dbcc0d95","Type":"ContainerStarted","Data":"9ee7f6c98e7ba128826088c8fee3992767668371581f842f6980946acc842e83"}
Sep 19 08:25:44.176104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:44.176084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:25:44.186720 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:44.186671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c2jqk" podStartSLOduration=2.899909913 podStartE2EDuration="4.186656403s" totalImagesPullingTime="1.28674649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:44.185843642 +0000 UTC m=+308.506515235" watchObservedRunningTime="2026-09-19 08:25:44.186656403 +0000 UTC m=+308.507327991"
Sep 19 08:25:45.043696 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:45.043661    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:45.094707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:45.094674    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:45.952643 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:45.952606    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.087654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.087603    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.195035 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.195002    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.204151 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.204081    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wzj4q"]
Sep 19 08:25:46.207268 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.207250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.209827 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.209809    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:25:46.209924 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.209846    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97wlz\""
Sep 19 08:25:46.210305 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.210291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:25:46.210385 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.210322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:25:46.265573 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.265547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.272540 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxhj\" (UniqueName: \"kubernetes.io/projected/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-kube-api-access-dmxhj\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-wtmp\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-root\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-sys\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-tls\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-metrics-client-ca\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.272846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.272741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-textfile\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-metrics-client-ca\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-textfile\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxhj\" (UniqueName: \"kubernetes.io/projected/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-kube-api-access-dmxhj\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-wtmp\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-root\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.373856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-sys\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-tls\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.373947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-wtmp\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-root\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-textfile\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-sys\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-metrics-client-ca\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374647 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.374817 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.374795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.376275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.376255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-tls\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.376488 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.376469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.381802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.381779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxhj\" (UniqueName: \"kubernetes.io/projected/ff7543c6-457a-43c4-8beb-32e8a9e76d9f-kube-api-access-dmxhj\") pod \"node-exporter-wzj4q\" (UID: \"ff7543c6-457a-43c4-8beb-32e8a9e76d9f\") " pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.473393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.473315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.516057 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.516031    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzj4q"
Sep 19 08:25:46.525952 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:46.525929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7543c6_457a_43c4_8beb_32e8a9e76d9f.slice/crio-6b79328cc245738e28465b994d1125e61d44974903c16686f8167be7894a7bad WatchSource:0}: Error finding container 6b79328cc245738e28465b994d1125e61d44974903c16686f8167be7894a7bad: Status 404 returned error can't find the container with id 6b79328cc245738e28465b994d1125e61d44974903c16686f8167be7894a7bad
Sep 19 08:25:46.574119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.574091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.737504 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.737424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:46.837984 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:46.837951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:47.179378 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:47.179349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:47.181973 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:47.181947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzj4q" event={"ID":"ff7543c6-457a-43c4-8beb-32e8a9e76d9f","Type":"ContainerStarted","Data":"6b79328cc245738e28465b994d1125e61d44974903c16686f8167be7894a7bad"}
Sep 19 08:25:47.284858 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:47.284816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:47.582574 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:47.582545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:48.186644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:48.186608    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff7543c6-457a-43c4-8beb-32e8a9e76d9f" containerID="e1b9247e58cfe47a12e93d915c9ea5fc6302491249cb8c3c17c97173ce81e032" exitCode=0
Sep 19 08:25:48.187052 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:48.186647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzj4q" event={"ID":"ff7543c6-457a-43c4-8beb-32e8a9e76d9f","Type":"ContainerDied","Data":"e1b9247e58cfe47a12e93d915c9ea5fc6302491249cb8c3c17c97173ce81e032"}
Sep 19 08:25:49.192239 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.192207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzj4q" event={"ID":"ff7543c6-457a-43c4-8beb-32e8a9e76d9f","Type":"ContainerStarted","Data":"a052c6becd8f1bdf6c537b12be975f726cec7edffe9197c839cbf4628d897cdb"}
Sep 19 08:25:49.192612 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.192267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzj4q" event={"ID":"ff7543c6-457a-43c4-8beb-32e8a9e76d9f","Type":"ContainerStarted","Data":"a01b3613afe48ca7eb8ea3ab64c0f62bcd4c2372f2c01bbb5821d849ebd198a1"}
Sep 19 08:25:49.211321 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.211262    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wzj4q" podStartSLOduration=2.234494481 podStartE2EDuration="3.211246545s" totalImagesPullingTime="917.261502ms" totalInitContainerRuntime="59.490562ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:49.21014543 +0000 UTC m=+313.530817010" watchObservedRunningTime="2026-09-19 08:25:49.211246545 +0000 UTC m=+313.531918130"
Sep 19 08:25:49.227635 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.227604    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:49.328216 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.328187    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:49.428447 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.428412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:49.530288 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:49.530194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:50.971647 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:50.971620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:51.213319 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:51.213280    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:52.470651 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.470614    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:52.476807 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.476781    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:25:52.484809 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.484781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.488248 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488224    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 19 08:25:52.488374 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 19 08:25:52.488436 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488390    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 19 08:25:52.488661 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 19 08:25:52.488708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 19 08:25:52.488757 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.488694    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 19 08:25:52.490091 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.490072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ncqtcs5sfna2\""
Sep 19 08:25:52.490278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.490094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 19 08:25:52.492797 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.492763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 19 08:25:52.492901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.492881    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 19 08:25:52.492979 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.492951    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f7n85\""
Sep 19 08:25:52.492979 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.492968    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 19 08:25:52.493084 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.492851    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 19 08:25:52.493237 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.493219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 19 08:25:52.493343 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.493326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 19 08:25:52.493444 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.493329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 19 08:25:52.495018 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.494999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 19 08:25:52.498205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.498185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:25:52.610928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.610897    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:52.622434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfbg\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.622991 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.622982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfbg\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.723825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724074 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724074 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724074 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.723989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724074 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724278 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724628 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724628 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724628 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.724796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.725006 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.724977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.726053 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.726026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.726526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.726503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.729120 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.728581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.729493 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.729424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.729580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.729528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.729580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.729572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.730088 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.730356 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.730452 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.730515 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.730957 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.731060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.731060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.730977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.731441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.731422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.731952 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.731931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.732601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.732582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfbg\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg\") pod \"prometheus-k8s-0\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:52.795685 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:52.795643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:25:53.083919 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.083892    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:25:53.086049 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:25:53.086019    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696eb8d1_081d_4f94_aa63_96f701f1877d.slice/crio-e499383208b5566c6dabceb0a29da0533c95185bafbef4468e053fab67adc7b5 WatchSource:0}: Error finding container e499383208b5566c6dabceb0a29da0533c95185bafbef4468e053fab67adc7b5: Status 404 returned error can't find the container with id e499383208b5566c6dabceb0a29da0533c95185bafbef4468e053fab67adc7b5
Sep 19 08:25:53.139033 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.139010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c5868599f-tqmnc"
Sep 19 08:25:53.206692 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.206661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"e499383208b5566c6dabceb0a29da0533c95185bafbef4468e053fab67adc7b5"}
Sep 19 08:25:53.275702 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.275673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:53.394402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.394367    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:53.492756 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:53.492717    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:54.116163 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:54.116130    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:54.216656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:54.216616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:55.216623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:55.216587    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9" exitCode=0
Sep 19 08:25:55.217011 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:55.216630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"}
Sep 19 08:25:55.838387 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:55.838341    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" podUID="4649de07-9935-42d6-bb85-e44884a90b14" containerName="registry" containerID="cri-o://4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63" gracePeriod=30
Sep 19 08:25:56.081631 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.081608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:56.155596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155565    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.155790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.155790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155650    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.155790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.155790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9skx\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.155790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.156040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.156040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.155830    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted\") pod \"4649de07-9935-42d6-bb85-e44884a90b14\" (UID: \"4649de07-9935-42d6-bb85-e44884a90b14\") "
Sep 19 08:25:56.156279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.156249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:56.157296 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.157255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:56.158353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.158324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:56.158784 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.158734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:56.159024 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.158969    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:56.159291 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.159268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx" (OuterVolumeSpecName: "kube-api-access-f9skx") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "kube-api-access-f9skx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:56.159532 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.159503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:56.167972 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.167941    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4649de07-9935-42d6-bb85-e44884a90b14" (UID: "4649de07-9935-42d6-bb85-e44884a90b14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:25:56.220952 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.220923    2566 generic.go:356] "Generic (PLEG): container finished" podID="4649de07-9935-42d6-bb85-e44884a90b14" containerID="4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63" exitCode=0
Sep 19 08:25:56.221316 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.220962    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd"
Sep 19 08:25:56.221316 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.221009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" event={"ID":"4649de07-9935-42d6-bb85-e44884a90b14","Type":"ContainerDied","Data":"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"}
Sep 19 08:25:56.221316 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.221070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9d78fcf6-lm4wd" event={"ID":"4649de07-9935-42d6-bb85-e44884a90b14","Type":"ContainerDied","Data":"ffaa365310202c0804ca127a9810642539b09214b0f7df3e9b837059b1302ba2"}
Sep 19 08:25:56.221316 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.221098    2566 scope.go:117] "RemoveContainer" containerID="4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"
Sep 19 08:25:56.228627 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.228603    2566 scope.go:117] "RemoveContainer" containerID="4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"
Sep 19 08:25:56.228989 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:25:56.228969    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63\": container with ID starting with 4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63 not found: ID does not exist" containerID="4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"
Sep 19 08:25:56.229061 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.228996    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63"} err="failed to get container status \"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63\": rpc error: code = NotFound desc = could not find container \"4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63\": container with ID starting with 4a3a199d7ff571fedbf1c40e4a1379d9ce93b59c4dc859e89769132d157aee63 not found: ID does not exist"
Sep 19 08:25:56.235853 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.235836    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:56.239472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.239455    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b9d78fcf6-lm4wd"]
Sep 19 08:25:56.256638 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256615    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-registry-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256638 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256635    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9skx\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-kube-api-access-f9skx\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256645    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-installation-pull-secrets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256654    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-registry-certificates\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256662    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4649de07-9935-42d6-bb85-e44884a90b14-ca-trust-extracted\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256670    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4649de07-9935-42d6-bb85-e44884a90b14-image-registry-private-configuration\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256678    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4649de07-9935-42d6-bb85-e44884a90b14-bound-sa-token\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:56.256748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:56.256686    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4649de07-9935-42d6-bb85-e44884a90b14-trusted-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:25:58.180762 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:58.180732    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4649de07-9935-42d6-bb85-e44884a90b14" path="/var/lib/kubelet/pods/4649de07-9935-42d6-bb85-e44884a90b14/volumes"
Sep 19 08:25:59.238017 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:59.237974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"}
Sep 19 08:25:59.239664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:25:59.239565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"}
Sep 19 08:26:00.829340 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:00.829306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:00.922930 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:00.922901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.023219 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.023189    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.122777 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.122741    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.223476 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.223442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.247366 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.247339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"}
Sep 19 08:26:01.247473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.247411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"}
Sep 19 08:26:01.247473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.247450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"}
Sep 19 08:26:01.247538 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.247488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerStarted","Data":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"}
Sep 19 08:26:01.278064 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.278018    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.30022978 podStartE2EDuration="9.278005952s" totalImagesPullingTime="5.960992623s" totalInitContainerRuntime="16.783549ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:01.276805539 +0000 UTC m=+325.597477126" watchObservedRunningTime="2026-09-19 08:26:01.278005952 +0000 UTC m=+325.598677540"
Sep 19 08:26:01.287127 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.287104    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.341137 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.341114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.501414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.501348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.555962 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.555935    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.919325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.919293    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:01.972419 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:01.972387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:02.125370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:02.125342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:02.181605 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:02.181530    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:02.512971 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:02.512895    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:02.796527 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:02.796458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:26:02.949674 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:02.949641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:03.228215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:03.228183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:03.937942 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:03.937912    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:04.042335 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:04.042300    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:07.145134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:07.145099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:07.246468 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:07.246439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:15.332788 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:15.332738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:16.185465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:16.185428    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:16.244548 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:16.244510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:24.157983 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:24.157950    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:24.258339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:24.258306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:27.330727 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:27.330695    2566 generic.go:356] "Generic (PLEG): container finished" podID="1cdd2703-c202-41de-acf8-46444e1feda6" containerID="1db283e7a36605b15389c9302937acefcf7146ec50679e419c6bb6898e43e79a" exitCode=0
Sep 19 08:26:27.331123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:27.330762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp" event={"ID":"1cdd2703-c202-41de-acf8-46444e1feda6","Type":"ContainerDied","Data":"1db283e7a36605b15389c9302937acefcf7146ec50679e419c6bb6898e43e79a"}
Sep 19 08:26:27.331123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:27.331058    2566 scope.go:117] "RemoveContainer" containerID="1db283e7a36605b15389c9302937acefcf7146ec50679e419c6bb6898e43e79a"
Sep 19 08:26:27.647949 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:27.647913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:28.336294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:28.336222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rnzpp" event={"ID":"1cdd2703-c202-41de-acf8-46444e1feda6","Type":"ContainerStarted","Data":"aa6a86e606ba57ec391f2ce4fddaeb1947f24220d64106ed2818a9fe61153427"}
Sep 19 08:26:29.252533 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:29.252505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:29.352782 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:29.352733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:29.758656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:29.758625    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:31.187099 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:31.187067    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:31.251474 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:31.251443    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:39.935315 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:39.935275    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:40.035993 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:40.035945    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:40.136031 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:40.135993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:46.189354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:46.189322    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:46.244377 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:46.244346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:49.945620 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:49.945580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:26:49.947819 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:49.947797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6df42b6-9d4c-4388-8750-2b7e19ccbe71-metrics-certs\") pod \"network-metrics-daemon-c4vdw\" (UID: \"f6df42b6-9d4c-4388-8750-2b7e19ccbe71\") " pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:26:50.179757 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:50.179733    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95gj6\""
Sep 19 08:26:50.187981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:50.187962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4vdw"
Sep 19 08:26:50.319471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:50.319442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.393283 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:50.393258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.458332 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:50.458264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4vdw"]
Sep 19 08:26:50.461032 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:26:50.461003    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6df42b6_9d4c_4388_8750_2b7e19ccbe71.slice/crio-ee7cf66e53868304aa1836078aa7765afea74380675689a3eb8948d8e0e55820 WatchSource:0}: Error finding container ee7cf66e53868304aa1836078aa7765afea74380675689a3eb8948d8e0e55820: Status 404 returned error can't find the container with id ee7cf66e53868304aa1836078aa7765afea74380675689a3eb8948d8e0e55820
Sep 19 08:26:51.412760 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:51.412738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4vdw" event={"ID":"f6df42b6-9d4c-4388-8750-2b7e19ccbe71","Type":"ContainerStarted","Data":"507b84e4688911bebbd24ecb1b0cd29220cfb7e3e65aeaf68455dabd9d740fd6"}
Sep 19 08:26:51.413026 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:51.412799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4vdw" event={"ID":"f6df42b6-9d4c-4388-8750-2b7e19ccbe71","Type":"ContainerStarted","Data":"ee7cf66e53868304aa1836078aa7765afea74380675689a3eb8948d8e0e55820"}
Sep 19 08:26:52.417946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.417900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4vdw" event={"ID":"f6df42b6-9d4c-4388-8750-2b7e19ccbe71","Type":"ContainerStarted","Data":"4abdde92113edced55d258b45df3a731635e7669bb53d7574cb8317d54bdab09"}
Sep 19 08:26:52.440279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.440240    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c4vdw" podStartSLOduration=375.64251743 podStartE2EDuration="6m16.440226371s" totalImagesPullingTime="797.708941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:52.439120111 +0000 UTC m=+376.759791699" watchObservedRunningTime="2026-09-19 08:26:52.440226371 +0000 UTC m=+376.760897958"
Sep 19 08:26:52.622902 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.622871    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:52.722005 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.721926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:52.796354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.796329    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:26:52.814431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:52.814410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:26:53.436501 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:26:53.436476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:03.898589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:03.898550    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:03.961753 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:03.961722    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:05.525497 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:05.525464    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:05.624844 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:05.624810    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:11.020407 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:11.020892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020786    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-web" containerID="cri-o://e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b" gracePeriod=600
Sep 19 08:27:11.020892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020778    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="config-reloader" containerID="cri-o://5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab" gracePeriod=600
Sep 19 08:27:11.020892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020812    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="thanos-sidecar" containerID="cri-o://0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115" gracePeriod=600
Sep 19 08:27:11.020892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020870    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy" containerID="cri-o://c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c" gracePeriod=600
Sep 19 08:27:11.021085 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020743    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="prometheus" containerID="cri-o://ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7" gracePeriod=600
Sep 19 08:27:11.021085 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.020823    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3" gracePeriod=600
Sep 19 08:27:11.265211 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.265183    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.405548 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405814    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.405907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjfbg\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405966    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.405998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406159    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle\") pod \"696eb8d1-081d-4f94-aa63-96f701f1877d\" (UID: \"696eb8d1-081d-4f94-aa63-96f701f1877d\") "
Sep 19 08:27:11.406760 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406508    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.406760 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406527    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.407912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.406995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.407912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.407072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.407912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.407296    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.407912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.407616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.407912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.407854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:11.408244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.408088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:11.408845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.408816    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.408946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.408870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config" (OuterVolumeSpecName: "config") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.409012 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.408987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:11.409461 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.409425    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.409738 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.409712    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.410077 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.410045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.410253 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.410222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg" (OuterVolumeSpecName: "kube-api-access-rjfbg") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "kube-api-access-rjfbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:11.410378 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.410360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out" (OuterVolumeSpecName: "config-out") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:11.410529 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.410500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.410654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.410639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.411083 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.411069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.422893 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.422867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config" (OuterVolumeSpecName: "web-config") pod "696eb8d1-081d-4f94-aa63-96f701f1877d" (UID: "696eb8d1-081d-4f94-aa63-96f701f1877d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:11.485892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485864    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3" exitCode=0
Sep 19 08:27:11.485892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485885    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c" exitCode=0
Sep 19 08:27:11.485892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485891    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b" exitCode=0
Sep 19 08:27:11.485892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485897    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115" exitCode=0
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485902    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab" exitCode=0
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485907    2566 generic.go:356] "Generic (PLEG): container finished" podID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7" exitCode=0
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485944    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.485942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"}
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"}
Sep 19 08:27:11.486112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"}
Sep 19 08:27:11.486341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"}
Sep 19 08:27:11.486341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486129    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.486341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"}
Sep 19 08:27:11.486341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"}
Sep 19 08:27:11.486341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.486202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"696eb8d1-081d-4f94-aa63-96f701f1877d","Type":"ContainerDied","Data":"e499383208b5566c6dabceb0a29da0533c95185bafbef4468e053fab67adc7b5"}
Sep 19 08:27:11.493739 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.493724    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.502457 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.502440    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.507596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507578    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-configmap-metrics-client-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507599    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507615    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507630    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-tls-assets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507641    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507652    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-grpc-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507665    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-kube-rbac-proxy\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507676    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfbg\" (UniqueName: \"kubernetes.io/projected/696eb8d1-081d-4f94-aa63-96f701f1877d-kube-api-access-rjfbg\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507688    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-db\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507701    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507714    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507727    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-web-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507740    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507752    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-metrics-client-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507790    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/696eb8d1-081d-4f94-aa63-96f701f1877d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507805    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/696eb8d1-081d-4f94-aa63-96f701f1877d-config-out\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507818    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.507994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.507831    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/696eb8d1-081d-4f94-aa63-96f701f1877d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:27:11.508509 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.508490    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.515875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.515854    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:11.516018 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.516002    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.521909 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.521878    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.527891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.526039    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:11.533656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.533637    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.539565 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.539548    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.539821 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.539799    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.539865 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.539831    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.539865 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.539854    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.540085 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.540070    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.540135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.540135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540103    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.540310 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.540292    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.540369 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540320    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.540369 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540342    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.540573 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.540557    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.540618 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540578    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.540618 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540597    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.540871 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.540854    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.540922 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540874    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.540922 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.540896    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.541095 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.541080    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.541152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541101    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.541152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541120    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.541371 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:27:11.541356    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.541406 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541374    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.541406 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541386    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.541592 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.541650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541593    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.541807 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541792    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.541856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.541807    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.542026 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542009    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.542092 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542027    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.542249 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542231    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.542292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542250    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.542463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542444    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.542503 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542464    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.542658 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542641    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.542716 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542659    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.542875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542861    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.542940 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.542878    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.543097 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.543139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543098    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.543301 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543284    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.543346 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543301    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.543513 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543495    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.543573 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543514    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.543721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543705    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.543798 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543722    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.543946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543930    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.544002 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.543947    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.544155 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544139    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.544200 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544156    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.544398 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544380    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.544442 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544398    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.544633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544617    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.544690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544634    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.544871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544842    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.544871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.544866    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.545100 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545081    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.545206 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545102    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.545312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.545376 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545314    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.545556 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545538    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.545599 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545557    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.545789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545754    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.545850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545791    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.546015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.545998    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.546061 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546016    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.546207 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546191    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.546263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546208    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.546432 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546416    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.546472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546432    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.546612 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546595    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.546673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546613    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.546851 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546837    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.546902 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.546851    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.547037 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547021    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.547093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547038    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.547237 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547222    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.547298 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547238    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.547427 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547411    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.547470 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547433    2566 scope.go:117] "RemoveContainer" containerID="de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"
Sep 19 08:27:11.547613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547597    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3"} err="failed to get container status \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": rpc error: code = NotFound desc = could not find container \"de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3\": container with ID starting with de4a42ddf79b0f6583ca970fa9ab5c5c48dabf254d391e65262e787ab00ec6e3 not found: ID does not exist"
Sep 19 08:27:11.547684 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547613    2566 scope.go:117] "RemoveContainer" containerID="c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"
Sep 19 08:27:11.547813 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547797    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c"} err="failed to get container status \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": rpc error: code = NotFound desc = could not find container \"c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c\": container with ID starting with c47cb7b2df9515c8dcf7ddb1a93917aaa1a186af4d5aa196db2578d7f2345b0c not found: ID does not exist"
Sep 19 08:27:11.547860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547813    2566 scope.go:117] "RemoveContainer" containerID="e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"
Sep 19 08:27:11.547996 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547981    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b"} err="failed to get container status \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": rpc error: code = NotFound desc = could not find container \"e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b\": container with ID starting with e9ce195e51007a4fa7e4ee240bd9297e7c031ca808fdb3504c0c079df181766b not found: ID does not exist"
Sep 19 08:27:11.548059 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.547997    2566 scope.go:117] "RemoveContainer" containerID="0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"
Sep 19 08:27:11.548180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548164    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115"} err="failed to get container status \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": rpc error: code = NotFound desc = could not find container \"0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115\": container with ID starting with 0d8d922f22f67ceb80feeb8f85646ca4c5090f67c5cdc50d61556250ce775115 not found: ID does not exist"
Sep 19 08:27:11.548242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548181    2566 scope.go:117] "RemoveContainer" containerID="5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"
Sep 19 08:27:11.548371 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548356    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab"} err="failed to get container status \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": rpc error: code = NotFound desc = could not find container \"5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab\": container with ID starting with 5f8d53034a1877c3618cff24247431b9d0f787019de1bbb41456ed07c30a7eab not found: ID does not exist"
Sep 19 08:27:11.548434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548372    2566 scope.go:117] "RemoveContainer" containerID="ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"
Sep 19 08:27:11.548547 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548532    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7"} err="failed to get container status \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": rpc error: code = NotFound desc = could not find container \"ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7\": container with ID starting with ec5814398c4846012b8609548fb6b924370087233f9cca2ea16535222add8dd7 not found: ID does not exist"
Sep 19 08:27:11.548587 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548548    2566 scope.go:117] "RemoveContainer" containerID="08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"
Sep 19 08:27:11.548728 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.548713    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9"} err="failed to get container status \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": rpc error: code = NotFound desc = could not find container \"08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9\": container with ID starting with 08417aa31ee87ec116c64d2df0c80e1d903ee79ce0536282c87add421d49a0c9 not found: ID does not exist"
Sep 19 08:27:11.555472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555448    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:11.555682 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555671    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="thanos-sidecar"
Sep 19 08:27:11.555721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555684    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="thanos-sidecar"
Sep 19 08:27:11.555721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555696    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-thanos"
Sep 19 08:27:11.555721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555702    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-thanos"
Sep 19 08:27:11.555721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555709    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="prometheus"
Sep 19 08:27:11.555721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555716    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="prometheus"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4649de07-9935-42d6-bb85-e44884a90b14" containerName="registry"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555728    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4649de07-9935-42d6-bb85-e44884a90b14" containerName="registry"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555734    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="config-reloader"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555739    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="config-reloader"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555744    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555759    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-web"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555776    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-web"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555784    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="init-config-reloader"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555790    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="init-config-reloader"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555825    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="thanos-sidecar"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555831    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="prometheus"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555837    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-web"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555843    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4649de07-9935-42d6-bb85-e44884a90b14" containerName="registry"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555848    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555854    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="config-reloader"
Sep 19 08:27:11.555875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.555859    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" containerName="kube-rbac-proxy-thanos"
Sep 19 08:27:11.561367 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.561354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.565566 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.565550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 19 08:27:11.565566 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.565557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 19 08:27:11.565719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.565551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 19 08:27:11.565719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.565611    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 19 08:27:11.565719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.565707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f7n85\""
Sep 19 08:27:11.566183 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 19 08:27:11.566183 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566133    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 19 08:27:11.566183 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566140    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 19 08:27:11.566183 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566158    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 19 08:27:11.566385 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566157    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 19 08:27:11.566580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566556    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ncqtcs5sfna2\""
Sep 19 08:27:11.566860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566837    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 19 08:27:11.566938 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566842    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 19 08:27:11.566938 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 19 08:27:11.566938 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.566935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 19 08:27:11.570562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.570543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 19 08:27:11.574180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.574165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 19 08:27:11.582465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.582447    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:11.709123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2sk\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-kube-api-access-dh2sk\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.709634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.709505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810225 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810225 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810225 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810331 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810367 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810416 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810416 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.810507 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.810438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811079 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2sk\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-kube-api-access-dh2sk\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811370 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.811705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.811467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.812109 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.812086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.813414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.813500 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.813500 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.813500 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.814204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.814204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.813916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08b6efbc-53c8-4cd9-8336-958777ba81b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.814204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.814115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.814386 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.814249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.815055 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.815030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.815669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.815642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.815953 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.815934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.816020 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.816009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.816080 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.816064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08b6efbc-53c8-4cd9-8336-958777ba81b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.816132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.816113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08b6efbc-53c8-4cd9-8336-958777ba81b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.822261 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.822242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2sk\" (UniqueName: \"kubernetes.io/projected/08b6efbc-53c8-4cd9-8336-958777ba81b1-kube-api-access-dh2sk\") pod \"prometheus-k8s-0\" (UID: \"08b6efbc-53c8-4cd9-8336-958777ba81b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:11.870353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:11.870331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:12.151366 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:12.151342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 19 08:27:12.152957 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:27:12.152929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b6efbc_53c8_4cd9_8336_958777ba81b1.slice/crio-00b4819becb6b8e8bb1d874719751b4d0e6852a8e304d40b52870859a63d4660 WatchSource:0}: Error finding container 00b4819becb6b8e8bb1d874719751b4d0e6852a8e304d40b52870859a63d4660: Status 404 returned error can't find the container with id 00b4819becb6b8e8bb1d874719751b4d0e6852a8e304d40b52870859a63d4660
Sep 19 08:27:12.180456 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:12.180433    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696eb8d1-081d-4f94-aa63-96f701f1877d" path="/var/lib/kubelet/pods/696eb8d1-081d-4f94-aa63-96f701f1877d/volumes"
Sep 19 08:27:12.491466 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:12.491440    2566 generic.go:356] "Generic (PLEG): container finished" podID="08b6efbc-53c8-4cd9-8336-958777ba81b1" containerID="9b76f7244f61b7aef15dbd062410496873f805ccbae882e366337a656d958c6b" exitCode=0
Sep 19 08:27:12.491572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:12.491530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerDied","Data":"9b76f7244f61b7aef15dbd062410496873f805ccbae882e366337a656d958c6b"}
Sep 19 08:27:12.491622 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:12.491595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"00b4819becb6b8e8bb1d874719751b4d0e6852a8e304d40b52870859a63d4660"}
Sep 19 08:27:13.480895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.480836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:13.498798 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"e2f3288bd86bde54ee3bbd0ab5654c208780c93c1cd9c0d73e9054638a3e7b51"}
Sep 19 08:27:13.498895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"2d5c7c1103a19e44f75e9405b7c5c5aadf0c345bc8c9970748711377eff2554e"}
Sep 19 08:27:13.498895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"e32532d8139ee3fa906e2386d1af421582ae7a3e41a673afc6982bcdb3a5420b"}
Sep 19 08:27:13.498957 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"c19aa9cf8a52be5acb1c4927e76b233a93bfa78fc57c3b81953890dfeb414b46"}
Sep 19 08:27:13.498957 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"163d03eebe243ebcc2000e2d4395e5ce7e7bd1a84e9701710b1edfa52e86f141"}
Sep 19 08:27:13.499014 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.498966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"08b6efbc-53c8-4cd9-8336-958777ba81b1","Type":"ContainerStarted","Data":"a89dfe1c62092188d8b7cad14de8e56b9766ae2b356447fe64dbb73b64f55a80"}
Sep 19 08:27:13.553977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.553931    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.536081604 podStartE2EDuration="2.553907569s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.825965ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:13.553156031 +0000 UTC m=+397.873827617" watchObservedRunningTime="2026-09-19 08:27:13.553907569 +0000 UTC m=+397.874579157"
Sep 19 08:27:13.646954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.645526    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:13.750043 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:13.749953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:14.214644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:14.214615    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:14.316483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:14.316460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:16.870810 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:16.870761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:27:21.453825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:21.453793    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:21.514778 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:21.514742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:24.511858 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:24.511830    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:24.612584 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:24.612546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:24.714304 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:24.714273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:41.475818 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:41.475755    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:55.045728 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:27:55.045682    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:11.870720 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:11.870641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:28:11.885106 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:11.885086    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:28:12.724230 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:12.724205    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 19 08:28:20.804268 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:20.804232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:20.861798 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:20.861732    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:21.419251 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:21.419213    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:21.521786 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:21.521738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:24.094646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:24.094582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:24.152300 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:24.152269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:32.805596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:32.805552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:32.864159 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:32.864127    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:34.810914 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:34.810885    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:34.919900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:34.919862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:35.056022 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:35.055999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:46.193360 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:46.193322    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:46.293995 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:28:46.293961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:03.543685 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:03.543630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:34.178556 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:34.178470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:34.279318 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:34.279274    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:34.383352 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:34.383325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:56.723286 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:56.723245    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:56.810147 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:29:56.810113    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:02.632379 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:02.632327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.018147 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.018057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.098678 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.098638    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.162732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.162698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.584663 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.584636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.652689 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.652634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:03.996845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:03.996751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:04.097806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:04.097759    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:04.198561 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:04.198529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:04.313234 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:04.313149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:04.414480 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:04.414451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:04.513560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:04.513531    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:54.741235 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:54.741200    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:30:55.842342 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:30:55.842312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:19.753898 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:19.753863    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:19.836807 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:19.836758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:19.904180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:19.904150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:20.115609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:20.115572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:20.307877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:20.307832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:20.408509 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:20.408426    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:20.509010 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:20.508976    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:20.609361 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:20.609307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:27.010282 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:27.010239    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:40.482316 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:40.482276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:40.614309 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:40.614273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:40.716103 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:40.716077    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:41.214384 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:41.214351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:41.274224 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:41.274197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:43.958062 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:43.958024    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:51.721323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:51.721283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:51.780297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:51.780265    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:51.832490 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:51.832463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:55.522401 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:31:55.522359    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:26.170736 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:26.170699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:37.137109 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:37.137032    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:37.216673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:37.216646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:37.278141 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:37.278115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:37.380685 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:37.380656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:37.516353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:37.516276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:38.131221 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:38.131192    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:38.187045 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:38.187020    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:32:53.627815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.627755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vxtpn"]
Sep 19 08:32:53.630754 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.630732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.633240 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.633221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:32:53.633336 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.633222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:32:53.634186 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.634170    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:32:53.634296 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.634187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:32:53.634296 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.634187    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:32:53.638651 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.638630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vxtpn"]
Sep 19 08:32:53.714018 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.713992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.714018 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.714019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbjw\" (UniqueName: \"kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.814512 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.814486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.814612 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.814514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbjw\" (UniqueName: \"kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.814854 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.814839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.822799 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.822757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbjw\" (UniqueName: \"kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw\") pod \"s3-init-vxtpn\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") " pod="kserve/s3-init-vxtpn"
Sep 19 08:32:53.939373 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:53.939309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vxtpn"
Sep 19 08:32:54.210429 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:54.210252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vxtpn"]
Sep 19 08:32:54.213108 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:32:54.213074    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2864471_f847_4b29_9fa1_b02b4d4bb698.slice/crio-ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854 WatchSource:0}: Error finding container ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854: Status 404 returned error can't find the container with id ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854
Sep 19 08:32:54.215199 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:54.215182    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:32:54.654629 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:54.654596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerStarted","Data":"ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854"}
Sep 19 08:32:57.669962 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:57.669930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerStarted","Data":"a0248fffcea032d74f00ffbcb3e49de4ede11463095979ba7a99b6a620822b93"}
Sep 19 08:32:59.679060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:59.678978    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerID="a0248fffcea032d74f00ffbcb3e49de4ede11463095979ba7a99b6a620822b93" exitCode=0
Sep 19 08:32:59.679060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:32:59.679041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerDied","Data":"a0248fffcea032d74f00ffbcb3e49de4ede11463095979ba7a99b6a620822b93"}
Sep 19 08:33:04.700150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:33:04.700115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerStarted","Data":"67876f3799ba156c800517b2ff4efd82aaefe45c012895c8dbc5a8af7c87f7ce"}
Sep 19 08:33:04.717150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:33:04.717091    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vxtpn" podStartSLOduration=2.383210761 podStartE2EDuration="11.717070678s" totalImagesPullingTime="7.444403746s" totalInitContainerRuntime="1.889456171s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:04.715625542 +0000 UTC m=+749.036297128" watchObservedRunningTime="2026-09-19 08:33:04.717070678 +0000 UTC m=+749.037742272"
Sep 19 08:33:04.891679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:33:04.891644    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:04.947457 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:33:04.947427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:08.922954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:08.922926    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerID="67876f3799ba156c800517b2ff4efd82aaefe45c012895c8dbc5a8af7c87f7ce" exitCode=0
Sep 19 08:34:08.923339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:08.922987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerDied","Data":"67876f3799ba156c800517b2ff4efd82aaefe45c012895c8dbc5a8af7c87f7ce"}
Sep 19 08:34:10.045608 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.045580    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vxtpn"
Sep 19 08:34:10.180940 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.180857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache\") pod \"a2864471-f847-4b29-9fa1-b02b4d4bb698\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") "
Sep 19 08:34:10.181067 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.180954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsbjw\" (UniqueName: \"kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw\") pod \"a2864471-f847-4b29-9fa1-b02b4d4bb698\" (UID: \"a2864471-f847-4b29-9fa1-b02b4d4bb698\") "
Sep 19 08:34:10.183066 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.183040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw" (OuterVolumeSpecName: "kube-api-access-xsbjw") pod "a2864471-f847-4b29-9fa1-b02b4d4bb698" (UID: "a2864471-f847-4b29-9fa1-b02b4d4bb698"). InnerVolumeSpecName "kube-api-access-xsbjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:10.202574 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.202550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2864471-f847-4b29-9fa1-b02b4d4bb698" (UID: "a2864471-f847-4b29-9fa1-b02b4d4bb698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:10.281572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.281540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsbjw\" (UniqueName: \"kubernetes.io/projected/a2864471-f847-4b29-9fa1-b02b4d4bb698-kube-api-access-xsbjw\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:34:10.281572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.281570    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2864471-f847-4b29-9fa1-b02b4d4bb698-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:34:10.930906 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.930875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vxtpn" event={"ID":"a2864471-f847-4b29-9fa1-b02b4d4bb698","Type":"ContainerDied","Data":"ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854"}
Sep 19 08:34:10.930906 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.930893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vxtpn"
Sep 19 08:34:10.930906 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:10.930913    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9425bdbb8c65ccd0911a483ff673089b062d242a5f0f6556deaa080cb4c854"
Sep 19 08:34:11.637970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:11.637942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:44.707483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707449    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-sh67h"]
Sep 19 08:34:44.707960 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707718    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerName="s3-init"
Sep 19 08:34:44.707960 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707730    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerName="s3-init"
Sep 19 08:34:44.707960 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707744    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerName="download-hf-model"
Sep 19 08:34:44.707960 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707750    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerName="download-hf-model"
Sep 19 08:34:44.707960 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.707830    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2864471-f847-4b29-9fa1-b02b4d4bb698" containerName="s3-init"
Sep 19 08:34:44.710499 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.710484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.713115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.713092    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:34:44.713232 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.713197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:34:44.713280 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.713261    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:34:44.713324 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.713275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 19 08:34:44.714123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.714106    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:34:44.717800 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.717780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sh67h"]
Sep 19 08:34:44.848719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.848684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.848926 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.848742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqfk\" (UniqueName: \"kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.949361 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.949326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqfk\" (UniqueName: \"kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.949489 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.949384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.949705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.949691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:44.958249 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:44.958197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqfk\" (UniqueName: \"kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk\") pod \"s3-tls-init-custom-sh67h\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") " pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:45.019475 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:45.019426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:45.287886 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:45.287863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sh67h"]
Sep 19 08:34:45.290233 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:34:45.290210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2aedc62_185d_438d_8a1b_55c8243125a9.slice/crio-eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1 WatchSource:0}: Error finding container eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1: Status 404 returned error can't find the container with id eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1
Sep 19 08:34:46.053968 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:46.053928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerStarted","Data":"f419eda9011967eb32c56f87b840eb81650f6aed90e5761bc6d5f9649ff93897"}
Sep 19 08:34:46.054411 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:46.053981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerStarted","Data":"eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1"}
Sep 19 08:34:47.058581 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:47.058495    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerID="f419eda9011967eb32c56f87b840eb81650f6aed90e5761bc6d5f9649ff93897" exitCode=0
Sep 19 08:34:47.058581 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:47.058566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerDied","Data":"f419eda9011967eb32c56f87b840eb81650f6aed90e5761bc6d5f9649ff93897"}
Sep 19 08:34:48.063724 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:48.063684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerStarted","Data":"3181019a4f259114da940e7d75d9d21932f5124d670d9b42ba7a89db8471d170"}
Sep 19 08:34:48.081025 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:48.080975    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-sh67h" podStartSLOduration=2.683316419 podStartE2EDuration="4.080958503s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.397642084s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:48.080157676 +0000 UTC m=+852.400829266" watchObservedRunningTime="2026-09-19 08:34:48.080958503 +0000 UTC m=+852.401630091"
Sep 19 08:34:54.089034 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:54.088996    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerID="3181019a4f259114da940e7d75d9d21932f5124d670d9b42ba7a89db8471d170" exitCode=0
Sep 19 08:34:54.089497 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:54.089057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerDied","Data":"3181019a4f259114da940e7d75d9d21932f5124d670d9b42ba7a89db8471d170"}
Sep 19 08:34:55.224886 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.224859    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:55.324061 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.324022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache\") pod \"c2aedc62-185d-438d-8a1b-55c8243125a9\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") "
Sep 19 08:34:55.324214 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.324108    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpqfk\" (UniqueName: \"kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk\") pod \"c2aedc62-185d-438d-8a1b-55c8243125a9\" (UID: \"c2aedc62-185d-438d-8a1b-55c8243125a9\") "
Sep 19 08:34:55.326087 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.326055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk" (OuterVolumeSpecName: "kube-api-access-gpqfk") pod "c2aedc62-185d-438d-8a1b-55c8243125a9" (UID: "c2aedc62-185d-438d-8a1b-55c8243125a9"). InnerVolumeSpecName "kube-api-access-gpqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:55.346693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.346631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2aedc62-185d-438d-8a1b-55c8243125a9" (UID: "c2aedc62-185d-438d-8a1b-55c8243125a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:55.425535 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.425507    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqfk\" (UniqueName: \"kubernetes.io/projected/c2aedc62-185d-438d-8a1b-55c8243125a9-kube-api-access-gpqfk\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:34:55.425535 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:55.425530    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2aedc62-185d-438d-8a1b-55c8243125a9-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:34:56.098192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:56.098151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sh67h" event={"ID":"c2aedc62-185d-438d-8a1b-55c8243125a9","Type":"ContainerDied","Data":"eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1"}
Sep 19 08:34:56.098192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:56.098188    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0f7d32c3ac199689f2e4c18e370d3ced2a147419c3687f3fac76cbff5c41a1"
Sep 19 08:34:56.098192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:56.098163    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sh67h"
Sep 19 08:34:56.997387 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:56.997337    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:57.217515 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:57.217481    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:58.993421 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-f5wgw"]
Sep 19 08:34:58.993912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerName="download-hf-model"
Sep 19 08:34:58.993912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993822    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerName="download-hf-model"
Sep 19 08:34:58.993912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993858    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerName="s3-tls-init-custom"
Sep 19 08:34:58.993912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993866    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerName="s3-tls-init-custom"
Sep 19 08:34:58.994113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.993922    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2aedc62-185d-438d-8a1b-55c8243125a9" containerName="s3-tls-init-custom"
Sep 19 08:34:58.997676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:58.997653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.000113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.000093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:34:59.000113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.000113    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:34:59.000269 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.000139    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:34:59.000914 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.000897    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:34:59.000997 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.000898    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 19 08:34:59.003295 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.003272    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f5wgw"]
Sep 19 08:34:59.054952 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.054916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchl5\" (UniqueName: \"kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.054952 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.054952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.155941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.155908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kchl5\" (UniqueName: \"kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.156082 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.155943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.156285 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.156269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.164732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.164704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchl5\" (UniqueName: \"kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5\") pod \"s3-tls-init-serving-f5wgw\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") " pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.306580 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.306489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:34:59.576472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:34:59.576276    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f5wgw"]
Sep 19 08:34:59.579058 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:34:59.579032    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ddf3167_2288_428b_9949_feddb1309009.slice/crio-c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018 WatchSource:0}: Error finding container c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018: Status 404 returned error can't find the container with id c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018
Sep 19 08:35:00.115154 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:00.115123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerStarted","Data":"6f456bcddd4e3403875517788f53b11cdd187b2ec8eee8d315f0a903af9ba650"}
Sep 19 08:35:00.115154 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:00.115160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerStarted","Data":"c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018"}
Sep 19 08:35:01.120462 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:01.120433    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ddf3167-2288-428b-9949-feddb1309009" containerID="6f456bcddd4e3403875517788f53b11cdd187b2ec8eee8d315f0a903af9ba650" exitCode=0
Sep 19 08:35:01.120814 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:01.120509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerDied","Data":"6f456bcddd4e3403875517788f53b11cdd187b2ec8eee8d315f0a903af9ba650"}
Sep 19 08:35:02.126063 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:02.126031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerStarted","Data":"68ee4760f8750aea512dbf8f43d9f6055014e2f3eaa10211e6c4037d6cf582c3"}
Sep 19 08:35:02.142783 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:02.142723    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-f5wgw" podStartSLOduration=2.71477612 podStartE2EDuration="4.142710008s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.427933888s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:02.141727746 +0000 UTC m=+866.462399333" watchObservedRunningTime="2026-09-19 08:35:02.142710008 +0000 UTC m=+866.463381596"
Sep 19 08:35:08.150781 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:08.150734    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ddf3167-2288-428b-9949-feddb1309009" containerID="68ee4760f8750aea512dbf8f43d9f6055014e2f3eaa10211e6c4037d6cf582c3" exitCode=0
Sep 19 08:35:08.151112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:08.150805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerDied","Data":"68ee4760f8750aea512dbf8f43d9f6055014e2f3eaa10211e6c4037d6cf582c3"}
Sep 19 08:35:09.278982 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.278957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:35:09.331604 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.331580    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kchl5\" (UniqueName: \"kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5\") pod \"4ddf3167-2288-428b-9949-feddb1309009\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") "
Sep 19 08:35:09.331717 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.331630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache\") pod \"4ddf3167-2288-428b-9949-feddb1309009\" (UID: \"4ddf3167-2288-428b-9949-feddb1309009\") "
Sep 19 08:35:09.333483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.333454    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5" (OuterVolumeSpecName: "kube-api-access-kchl5") pod "4ddf3167-2288-428b-9949-feddb1309009" (UID: "4ddf3167-2288-428b-9949-feddb1309009"). InnerVolumeSpecName "kube-api-access-kchl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:35:09.356005 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.355968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ddf3167-2288-428b-9949-feddb1309009" (UID: "4ddf3167-2288-428b-9949-feddb1309009"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:35:09.432983 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.432928    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddf3167-2288-428b-9949-feddb1309009-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:35:09.432983 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:09.432951    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchl5\" (UniqueName: \"kubernetes.io/projected/4ddf3167-2288-428b-9949-feddb1309009-kube-api-access-kchl5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:35:10.159820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:10.159785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f5wgw" event={"ID":"4ddf3167-2288-428b-9949-feddb1309009","Type":"ContainerDied","Data":"c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018"}
Sep 19 08:35:10.159820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:10.159805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f5wgw"
Sep 19 08:35:10.160015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:10.159831    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7efd32d44a16eeabb6798e82d37afbd872f38670d8d43eed6befe7e1b2c8018"
Sep 19 08:35:10.681449 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:10.681413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:10.783320 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:10.783295    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:23.263167 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.263128    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:23.320684 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.320653    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:35:23.321019 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.321001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddf3167-2288-428b-9949-feddb1309009" containerName="s3-tls-init-serving"
Sep 19 08:35:23.321019 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.321020    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddf3167-2288-428b-9949-feddb1309009" containerName="s3-tls-init-serving"
Sep 19 08:35:23.321134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.321042    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddf3167-2288-428b-9949-feddb1309009" containerName="download-hf-model"
Sep 19 08:35:23.321134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.321048    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddf3167-2288-428b-9949-feddb1309009" containerName="download-hf-model"
Sep 19 08:35:23.321134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.321087    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddf3167-2288-428b-9949-feddb1309009" containerName="s3-tls-init-serving"
Sep 19 08:35:23.324844 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.324823    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.328140 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:35:23.328140 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 19 08:35:23.328354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 19 08:35:23.328354 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328111    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:35:23.328467 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328391    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:35:23.328519 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:35:23.328575 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.328537    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:35:23.334562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.334533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:35:23.435116 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.435085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.435294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.435122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mm5z\" (UniqueName: \"kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.435294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.435154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.435414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.435284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.435414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.435325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.462359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.462324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:23.536700 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.536615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.536700 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.536657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.536700 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.536699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.536984 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.536729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mm5z\" (UniqueName: \"kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.536984 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.536803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.537060 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:35:23.537041    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 19 08:35:23.537127 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:35:23.537115    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls podName:21c07050-8a62-41a2-a4c5-25614c9ea76a nodeName:}" failed. No retries permitted until 2026-09-19 08:35:24.037094342 +0000 UTC m=+888.357765913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls") pod "isvc-sklearn-batcher-predictor-78888c66b-sp7mx" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 19 08:35:23.537186 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.537124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.537470 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.537454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.537541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.537525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.545212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.545184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mm5z\" (UniqueName: \"kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:23.618139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.618101    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:23.860150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:23.860118    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:24.040570 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:24.040529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:24.042900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:24.042873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-sp7mx\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:24.255869 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:24.255760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:24.529714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:24.529639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:35:24.533057 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:35:24.533034    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c07050_8a62_41a2_a4c5_25614c9ea76a.slice/crio-a2d9c1774e44fb127fb731b46cb3d6fa191c29c2187180a0503298538e346bd9 WatchSource:0}: Error finding container a2d9c1774e44fb127fb731b46cb3d6fa191c29c2187180a0503298538e346bd9: Status 404 returned error can't find the container with id a2d9c1774e44fb127fb731b46cb3d6fa191c29c2187180a0503298538e346bd9
Sep 19 08:35:25.216606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:25.216571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerStarted","Data":"b84c4637f2a15bbc070b54cc379d4c24c01b64cc5026234322b74d1ecdd2e515"}
Sep 19 08:35:25.216812 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:25.216633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerStarted","Data":"a2d9c1774e44fb127fb731b46cb3d6fa191c29c2187180a0503298538e346bd9"}
Sep 19 08:35:28.229679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:28.229649    2566 generic.go:356] "Generic (PLEG): container finished" podID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerID="b84c4637f2a15bbc070b54cc379d4c24c01b64cc5026234322b74d1ecdd2e515" exitCode=0
Sep 19 08:35:28.230055 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:28.229724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerDied","Data":"b84c4637f2a15bbc070b54cc379d4c24c01b64cc5026234322b74d1ecdd2e515"}
Sep 19 08:35:28.392302 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:28.392269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:28.453939 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:28.453866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:33.175799 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:33.175748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:38.370023 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:38.369981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:38.590964 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:38.590905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:42.301055 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:42.301017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerStarted","Data":"9b101b082aafb29621f672cc37735db59d0af167a0c1d0e094dfa844955cf3e4"}
Sep 19 08:35:44.311291 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:44.311215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerStarted","Data":"19ff094e9facf698093c35f86b8c9a37457891c826fcf1a8a3ba733519e02a17"}
Sep 19 08:35:46.320455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:46.320417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerStarted","Data":"13200c680199633152d6655de2232feb20ec00097c6f389c8c659f32f566c0c0"}
Sep 19 08:35:46.320860 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:46.320520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:46.343356 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:46.343314    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podStartSLOduration=2.272998042 podStartE2EDuration="23.343300847s" totalImagesPullingTime="17.515200974s" totalInitContainerRuntime="3.555101831s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:46.341797244 +0000 UTC m=+910.662468830" watchObservedRunningTime="2026-09-19 08:35:46.343300847 +0000 UTC m=+910.663972435"
Sep 19 08:35:47.324733 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:47.324691    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:47.325150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:47.324789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:47.326139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:47.326110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:35:47.326953 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:47.326931    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:35:48.328677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:48.328623    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:35:48.329187 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:48.329152    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:35:48.332679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:48.332660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:35:49.333197 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:49.333157    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:35:49.333641 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:49.333534    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:35:59.333812 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:59.333744    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:35:59.338481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:35:59.334230    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:09.333152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:09.333105    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:36:09.333674 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:09.333645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:19.333821 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:19.333757    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:36:19.334260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:19.334235    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:29.334185 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:29.334137    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:36:29.334697 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:29.334527    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:39.333817 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:39.333752    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:36:39.334274 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:39.334249    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:49.333279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:49.333228    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:36:49.333810 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:49.333732    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:36:59.333665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:59.333578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:36:59.334118 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:36:59.333896    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:08.708256 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.708215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:37:08.708887 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.708680    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" containerID="cri-o://9b101b082aafb29621f672cc37735db59d0af167a0c1d0e094dfa844955cf3e4" gracePeriod=30
Sep 19 08:37:08.709006 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.708960    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" containerID="cri-o://13200c680199633152d6655de2232feb20ec00097c6f389c8c659f32f566c0c0" gracePeriod=30
Sep 19 08:37:08.709521 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.709160    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" containerID="cri-o://19ff094e9facf698093c35f86b8c9a37457891c826fcf1a8a3ba733519e02a17" gracePeriod=30
Sep 19 08:37:08.751629 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.751595    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:08.840321 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.840292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:37:08.843600 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.843580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.846097 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.846077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 19 08:37:08.846189 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.846152    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 19 08:37:08.859270 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.859248    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:37:08.888921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.888890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.889093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.888931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.889093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.888971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckrm\" (UniqueName: \"kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.889093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.889038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.889093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.889067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.989981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.989884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.989981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.989946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990217 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:37:08.990011    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 19 08:37:08.990217 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:37:08.990096    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls podName:836fe097-86aa-42af-b0df-68cc5062fb83 nodeName:}" failed. No retries permitted until 2026-09-19 08:37:09.490074571 +0000 UTC m=+993.810746137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 19 08:37:08.990217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckrm\" (UniqueName: \"kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990450 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990687 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.990806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.990788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:08.998309 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:08.998278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckrm\" (UniqueName: \"kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:09.333478 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.333381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:37:09.333791 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.333736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:09.493919 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.493878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:09.496439 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.496404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:09.639550 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.639513    2566 generic.go:356] "Generic (PLEG): container finished" podID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerID="19ff094e9facf698093c35f86b8c9a37457891c826fcf1a8a3ba733519e02a17" exitCode=2
Sep 19 08:37:09.639725 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.639573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerDied","Data":"19ff094e9facf698093c35f86b8c9a37457891c826fcf1a8a3ba733519e02a17"}
Sep 19 08:37:09.753324 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.753284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:09.773656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:09.773620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:10.027837 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:10.027743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:37:10.031376 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:37:10.031344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836fe097_86aa_42af_b0df_68cc5062fb83.slice/crio-4105c1e935be73891c94943612094b61d3f819d7b5f02ead36ad370c905191a0 WatchSource:0}: Error finding container 4105c1e935be73891c94943612094b61d3f819d7b5f02ead36ad370c905191a0: Status 404 returned error can't find the container with id 4105c1e935be73891c94943612094b61d3f819d7b5f02ead36ad370c905191a0
Sep 19 08:37:10.645517 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:10.645481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerStarted","Data":"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"}
Sep 19 08:37:10.645682 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:10.645548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerStarted","Data":"4105c1e935be73891c94943612094b61d3f819d7b5f02ead36ad370c905191a0"}
Sep 19 08:37:13.329677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:13.329637    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:14.664197 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:14.664163    2566 generic.go:356] "Generic (PLEG): container finished" podID="836fe097-86aa-42af-b0df-68cc5062fb83" containerID="b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5" exitCode=0
Sep 19 08:37:14.664632 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:14.664245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerDied","Data":"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"}
Sep 19 08:37:14.668367 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:14.668336    2566 generic.go:356] "Generic (PLEG): container finished" podID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerID="9b101b082aafb29621f672cc37735db59d0af167a0c1d0e094dfa844955cf3e4" exitCode=0
Sep 19 08:37:14.668512 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:14.668469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerDied","Data":"9b101b082aafb29621f672cc37735db59d0af167a0c1d0e094dfa844955cf3e4"}
Sep 19 08:37:15.675276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.675240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerStarted","Data":"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"}
Sep 19 08:37:15.675633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.675311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerStarted","Data":"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"}
Sep 19 08:37:15.675633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.675336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerStarted","Data":"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"}
Sep 19 08:37:15.675633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.675604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:15.675830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.675658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:15.677049 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.677024    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:15.695143 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:15.695103    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podStartSLOduration=3.792202185 podStartE2EDuration="7.695085724s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.902883539s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:37:15.694023405 +0000 UTC m=+1000.014695005" watchObservedRunningTime="2026-09-19 08:37:15.695085724 +0000 UTC m=+1000.015757294"
Sep 19 08:37:16.679813 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:16.679784    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:16.680282 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:16.679930    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:16.681076 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:16.681052    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:17.684367 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:17.684322    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:17.684824 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:17.684803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:18.329313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:18.329270    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:19.333406 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:19.333351    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:37:19.333856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:19.333710    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:22.689123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:22.689092    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:37:22.689614 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:22.689586    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:22.690086 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:22.690057    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:23.328855 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:23.328803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:23.329040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:23.328961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:28.329058 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:28.329007    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:28.936206 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:28.936167    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:29.333826 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:29.333696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 19 08:37:29.334250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:29.333878    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:29.334250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:29.334099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:29.334250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:29.334174    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:32.690135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:32.690093    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:32.690505 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:32.690471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:33.329827 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:33.329758    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:34.107087 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:34.107051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:34.270646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:34.270607    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:38.329232 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.329193    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 19 08:37:38.779788 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.779740    2566 generic.go:356] "Generic (PLEG): container finished" podID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerID="13200c680199633152d6655de2232feb20ec00097c6f389c8c659f32f566c0c0" exitCode=0
Sep 19 08:37:38.779932 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.779815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerDied","Data":"13200c680199633152d6655de2232feb20ec00097c6f389c8c659f32f566c0c0"}
Sep 19 08:37:38.860412 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.860382    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:38.933381 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location\") pod \"21c07050-8a62-41a2-a4c5-25614c9ea76a\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") "
Sep 19 08:37:38.933564 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert\") pod \"21c07050-8a62-41a2-a4c5-25614c9ea76a\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") "
Sep 19 08:37:38.933564 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mm5z\" (UniqueName: \"kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z\") pod \"21c07050-8a62-41a2-a4c5-25614c9ea76a\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") "
Sep 19 08:37:38.933564 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config\") pod \"21c07050-8a62-41a2-a4c5-25614c9ea76a\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") "
Sep 19 08:37:38.933564 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933503    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") pod \"21c07050-8a62-41a2-a4c5-25614c9ea76a\" (UID: \"21c07050-8a62-41a2-a4c5-25614c9ea76a\") "
Sep 19 08:37:38.933789 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21c07050-8a62-41a2-a4c5-25614c9ea76a" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:38.933923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21c07050-8a62-41a2-a4c5-25614c9ea76a" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:38.933923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.933897    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21c07050-8a62-41a2-a4c5-25614c9ea76a" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:38.935839 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.935812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z" (OuterVolumeSpecName: "kube-api-access-7mm5z") pod "21c07050-8a62-41a2-a4c5-25614c9ea76a" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a"). InnerVolumeSpecName "kube-api-access-7mm5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:38.935839 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:38.935823    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21c07050-8a62-41a2-a4c5-25614c9ea76a" (UID: "21c07050-8a62-41a2-a4c5-25614c9ea76a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:39.034637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.034537    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:37:39.034637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.034573    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c07050-8a62-41a2-a4c5-25614c9ea76a-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:37:39.034637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.034586    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c07050-8a62-41a2-a4c5-25614c9ea76a-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:37:39.034637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.034598    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c07050-8a62-41a2-a4c5-25614c9ea76a-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:37:39.034637 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.034612    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mm5z\" (UniqueName: \"kubernetes.io/projected/21c07050-8a62-41a2-a4c5-25614c9ea76a-kube-api-access-7mm5z\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:37:39.786856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.786822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx" event={"ID":"21c07050-8a62-41a2-a4c5-25614c9ea76a","Type":"ContainerDied","Data":"a2d9c1774e44fb127fb731b46cb3d6fa191c29c2187180a0503298538e346bd9"}
Sep 19 08:37:39.786856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.786843    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"
Sep 19 08:37:39.787333 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.786907    2566 scope.go:117] "RemoveContainer" containerID="13200c680199633152d6655de2232feb20ec00097c6f389c8c659f32f566c0c0"
Sep 19 08:37:39.795489 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.795469    2566 scope.go:117] "RemoveContainer" containerID="19ff094e9facf698093c35f86b8c9a37457891c826fcf1a8a3ba733519e02a17"
Sep 19 08:37:39.802680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.802660    2566 scope.go:117] "RemoveContainer" containerID="9b101b082aafb29621f672cc37735db59d0af167a0c1d0e094dfa844955cf3e4"
Sep 19 08:37:39.809578 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.809556    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:37:39.810314 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.810298    2566 scope.go:117] "RemoveContainer" containerID="b84c4637f2a15bbc070b54cc379d4c24c01b64cc5026234322b74d1ecdd2e515"
Sep 19 08:37:39.814799 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:39.814759    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-sp7mx"]
Sep 19 08:37:40.180735 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:40.180703    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" path="/var/lib/kubelet/pods/21c07050-8a62-41a2-a4c5-25614c9ea76a/volumes"
Sep 19 08:37:42.690529 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:42.690484    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:42.691016 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:42.690931    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:51.899700 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:51.899664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:52.690361 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:52.690314    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:37:52.690711 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:37:52.690684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:02.690213 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:02.690154    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:02.690620 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:02.690592    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:12.689745 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:12.689701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:12.690268 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:12.690200    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:22.690263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:22.690208    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:22.690686 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:22.690599    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:25.176920 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:25.176862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:25.177412 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:25.177390    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:35.177989 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:35.177907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:38:35.178364 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:35.178004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:38:44.407996 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.407954    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:38:44.408679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.408238    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" containerID="cri-o://26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd" gracePeriod=30
Sep 19 08:38:44.408679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.408297    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" containerID="cri-o://5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17" gracePeriod=30
Sep 19 08:38:44.408679 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.408303    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" containerID="cri-o://e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d" gracePeriod=30
Sep 19 08:38:44.419325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.419290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:44.509634 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.509590    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:44.696592 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:44.696500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:45.054242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:45.054158    2566 generic.go:356] "Generic (PLEG): container finished" podID="836fe097-86aa-42af-b0df-68cc5062fb83" containerID="e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d" exitCode=2
Sep 19 08:38:45.054242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:45.054214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerDied","Data":"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"}
Sep 19 08:38:45.177260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:45.177220    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:45.177572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:45.177546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:47.685524 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:47.685471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:38:50.075615 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:50.075584    2566 generic.go:356] "Generic (PLEG): container finished" podID="836fe097-86aa-42af-b0df-68cc5062fb83" containerID="26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd" exitCode=0
Sep 19 08:38:50.076011 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:50.075659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerDied","Data":"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"}
Sep 19 08:38:52.685291 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:52.685246    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:38:53.673270 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:53.673237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:55.177366 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:55.177318    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:38:55.177834 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:55.177714    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:57.684978 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:57.684926    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:38:57.685402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:38:57.685079    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:39:02.685538 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:02.685494    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:39:04.495164 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.495125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:04.531098 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531061    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:39:04.531350 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531337    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy"
Sep 19 08:39:04.531393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531352    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy"
Sep 19 08:39:04.531393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531364    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="storage-initializer"
Sep 19 08:39:04.531393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="storage-initializer"
Sep 19 08:39:04.531393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531378    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container"
Sep 19 08:39:04.531393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container"
Sep 19 08:39:04.531541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531400    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent"
Sep 19 08:39:04.531541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531405    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent"
Sep 19 08:39:04.531541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531468    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kube-rbac-proxy"
Sep 19 08:39:04.531541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531475    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="kserve-container"
Sep 19 08:39:04.531541 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.531483    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c07050-8a62-41a2-a4c5-25614c9ea76a" containerName="agent"
Sep 19 08:39:04.534583 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.534564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.536802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.536776    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 19 08:39:04.537093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.537072    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 19 08:39:04.545684 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.545661    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:39:04.639650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.639617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.639650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.639654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.639882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.639681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.639882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.639699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb55\" (UniqueName: \"kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.639882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.639784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.645381 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.645351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:04.730544 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.730510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:04.740654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.740623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.740801 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.740660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.740838 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.740821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.740886 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.740860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb55\" (UniqueName: \"kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.740927 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.740909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.741070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.741053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.741528 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.741508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.741612 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.741536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.743461 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.743439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.750263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.750196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb55\" (UniqueName: \"kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55\") pod \"isvc-logger-predictor-76f676cbf7-82hv4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:04.844281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:04.844241    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:05.124699 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.124626    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:39:05.127251 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:39:05.127223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9d0542_d6ad_4e1d_b1fa_5bb5e496c5e4.slice/crio-45bdf20b59a97e962bc10f3d6e3d1026ef8c4d0b0898632b2020ba761a8350eb WatchSource:0}: Error finding container 45bdf20b59a97e962bc10f3d6e3d1026ef8c4d0b0898632b2020ba761a8350eb: Status 404 returned error can't find the container with id 45bdf20b59a97e962bc10f3d6e3d1026ef8c4d0b0898632b2020ba761a8350eb
Sep 19 08:39:05.139056 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.139021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerStarted","Data":"45bdf20b59a97e962bc10f3d6e3d1026ef8c4d0b0898632b2020ba761a8350eb"}
Sep 19 08:39:05.177591 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.177539    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:5000: connect: connection refused"
Sep 19 08:39:05.177816 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.177703    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:39:05.177893 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.177829    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:05.177970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:05.177955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:39:06.145114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:06.145074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerStarted","Data":"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"}
Sep 19 08:39:07.684722 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:07.684680    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:39:09.158122 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:09.158084    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerID="d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0" exitCode=0
Sep 19 08:39:09.158512 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:09.158134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerDied","Data":"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"}
Sep 19 08:39:09.750178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:09.750138    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:10.103763 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.103670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:10.165333 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerStarted","Data":"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"}
Sep 19 08:39:10.165730 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerStarted","Data":"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"}
Sep 19 08:39:10.165730 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerStarted","Data":"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"}
Sep 19 08:39:10.165838 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:10.165891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:10.165891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.165889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:10.167411 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.167368    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:10.168062 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.168036    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:10.189133 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:10.189083    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podStartSLOduration=2.319802093 podStartE2EDuration="6.189066919s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869264826s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:39:10.187922712 +0000 UTC m=+1114.508594305" watchObservedRunningTime="2026-09-19 08:39:10.189066919 +0000 UTC m=+1114.509738507"
Sep 19 08:39:11.169627 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:11.169582    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:11.170181 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:11.170158    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:12.684758 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:12.684712    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 19 08:39:14.550578 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.550545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:39:14.622583 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config\") pod \"836fe097-86aa-42af-b0df-68cc5062fb83\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") "
Sep 19 08:39:14.622736 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622639    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert\") pod \"836fe097-86aa-42af-b0df-68cc5062fb83\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") "
Sep 19 08:39:14.622736 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location\") pod \"836fe097-86aa-42af-b0df-68cc5062fb83\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") "
Sep 19 08:39:14.622736 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") pod \"836fe097-86aa-42af-b0df-68cc5062fb83\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") "
Sep 19 08:39:14.622736 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622722    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lckrm\" (UniqueName: \"kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm\") pod \"836fe097-86aa-42af-b0df-68cc5062fb83\" (UID: \"836fe097-86aa-42af-b0df-68cc5062fb83\") "
Sep 19 08:39:14.622981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "836fe097-86aa-42af-b0df-68cc5062fb83" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:14.623042 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.622986    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:39:14.623100 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.623043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "836fe097-86aa-42af-b0df-68cc5062fb83" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:39:14.623100 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.623068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "836fe097-86aa-42af-b0df-68cc5062fb83" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:14.624924 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.624903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "836fe097-86aa-42af-b0df-68cc5062fb83" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:14.625030 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.624954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm" (OuterVolumeSpecName: "kube-api-access-lckrm") pod "836fe097-86aa-42af-b0df-68cc5062fb83" (UID: "836fe097-86aa-42af-b0df-68cc5062fb83"). InnerVolumeSpecName "kube-api-access-lckrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:14.723654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.723546    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836fe097-86aa-42af-b0df-68cc5062fb83-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:39:14.723654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.723587    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836fe097-86aa-42af-b0df-68cc5062fb83-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:39:14.723654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.723602    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/836fe097-86aa-42af-b0df-68cc5062fb83-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:39:14.723654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:14.723614    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckrm\" (UniqueName: \"kubernetes.io/projected/836fe097-86aa-42af-b0df-68cc5062fb83-kube-api-access-lckrm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:39:15.187958 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.187924    2566 generic.go:356] "Generic (PLEG): container finished" podID="836fe097-86aa-42af-b0df-68cc5062fb83" containerID="5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17" exitCode=0
Sep 19 08:39:15.188275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.188006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"
Sep 19 08:39:15.188275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.188012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerDied","Data":"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"}
Sep 19 08:39:15.188275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.188080    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc" event={"ID":"836fe097-86aa-42af-b0df-68cc5062fb83","Type":"ContainerDied","Data":"4105c1e935be73891c94943612094b61d3f819d7b5f02ead36ad370c905191a0"}
Sep 19 08:39:15.188275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.188112    2566 scope.go:117] "RemoveContainer" containerID="5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"
Sep 19 08:39:15.199165 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.199070    2566 scope.go:117] "RemoveContainer" containerID="e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"
Sep 19 08:39:15.207379 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.207356    2566 scope.go:117] "RemoveContainer" containerID="26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"
Sep 19 08:39:15.215211 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.215191    2566 scope.go:117] "RemoveContainer" containerID="b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"
Sep 19 08:39:15.216570 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.216551    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:39:15.221945 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.221918    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-7sqkc"]
Sep 19 08:39:15.223359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.223340    2566 scope.go:117] "RemoveContainer" containerID="5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"
Sep 19 08:39:15.223655 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:39:15.223636    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17\": container with ID starting with 5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17 not found: ID does not exist" containerID="5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"
Sep 19 08:39:15.223707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.223667    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17"} err="failed to get container status \"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17\": rpc error: code = NotFound desc = could not find container \"5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17\": container with ID starting with 5b99210cecbd1c02e4d728c69f371625460fcf15448509edbcbfd3e31e979f17 not found: ID does not exist"
Sep 19 08:39:15.223707 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.223687    2566 scope.go:117] "RemoveContainer" containerID="e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"
Sep 19 08:39:15.223945 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:39:15.223919    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d\": container with ID starting with e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d not found: ID does not exist" containerID="e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"
Sep 19 08:39:15.223994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.223951    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d"} err="failed to get container status \"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d\": rpc error: code = NotFound desc = could not find container \"e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d\": container with ID starting with e6e8c49dbcaa3663597fefbdf0bd7adfb98d78f6bbf10cb82ccb32808958ca8d not found: ID does not exist"
Sep 19 08:39:15.223994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.223966    2566 scope.go:117] "RemoveContainer" containerID="26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"
Sep 19 08:39:15.224259 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:39:15.224241    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd\": container with ID starting with 26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd not found: ID does not exist" containerID="26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"
Sep 19 08:39:15.224323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.224269    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd"} err="failed to get container status \"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd\": rpc error: code = NotFound desc = could not find container \"26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd\": container with ID starting with 26f6d32156e47989a5330715d8145e86dfc80b8b03140588e198de18acbeb7bd not found: ID does not exist"
Sep 19 08:39:15.224323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.224292    2566 scope.go:117] "RemoveContainer" containerID="b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"
Sep 19 08:39:15.224548 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:39:15.224533    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5\": container with ID starting with b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5 not found: ID does not exist" containerID="b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"
Sep 19 08:39:15.224589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:15.224551    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5"} err="failed to get container status \"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5\": rpc error: code = NotFound desc = could not find container \"b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5\": container with ID starting with b4a94c5ecd292286702927e0b160f2e69d94b457c2b3c25700190b9d71ddb3a5 not found: ID does not exist"
Sep 19 08:39:16.174397 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:16.174365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:39:16.175021 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:16.174993    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:16.175439 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:16.175414    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:16.180118 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:16.180093    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" path="/var/lib/kubelet/pods/836fe097-86aa-42af-b0df-68cc5062fb83/volumes"
Sep 19 08:39:26.175809 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:26.175751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:26.176386 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:26.176356    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:36.175867 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:36.175818    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:36.176380 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:36.176297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:46.175576 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:46.175530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:46.176108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:46.176074    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:56.175778 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:56.175728    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:39:56.176301 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:39:56.176164    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:40:06.175031 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:06.174936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:40:06.175508 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:06.175391    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:40:16.175327 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:16.175287    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:40:16.175839 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:16.175810    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:40:20.176817 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:20.176750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:40:20.177279 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:20.177156    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:40:30.180918 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:30.180888    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:40:30.181489 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:30.180968    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:40:30.504529 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:30.504453    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:35.642454 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:35.642419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:36.000853 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:36.000740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:40.234362 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.234331    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:40:40.234783 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.234580    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" containerID="cri-o://5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f" gracePeriod=30
Sep 19 08:40:40.234783 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.234600    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" containerID="cri-o://dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11" gracePeriod=30
Sep 19 08:40:40.234783 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.234601    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" containerID="cri-o://20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3" gracePeriod=30
Sep 19 08:40:40.261483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.261449    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:40.363390 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.363354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:40.438212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.438174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:40.498375 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.498294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:40.516650 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.516621    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerID="dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11" exitCode=2
Sep 19 08:40:40.516796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:40.516670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerDied","Data":"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"}
Sep 19 08:40:41.170193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:41.170149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:40:45.541570 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:45.541536    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerID="5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f" exitCode=0
Sep 19 08:40:45.542103 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:45.541609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerDied","Data":"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"}
Sep 19 08:40:46.170293 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:46.170252    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:40:50.177124 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:50.177078    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:40:50.177544 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:50.177459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:40:51.170324 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:51.170284    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:40:51.170509 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:51.170423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:40:56.170651 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:40:56.170606    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:41:00.177414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:00.177361    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:41:00.179844 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:00.177685    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:41:01.170710 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:01.170670    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:41:06.170717 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:06.170679    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 19 08:41:10.177256 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.177213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 19 08:41:10.177824 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.177788    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:41:10.180558 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.180537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:41:10.180680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.180629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:41:10.373710 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.373683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:41:10.490588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location\") pod \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") "
Sep 19 08:41:10.490588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config\") pod \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") "
Sep 19 08:41:10.490806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqb55\" (UniqueName: \"kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55\") pod \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") "
Sep 19 08:41:10.490806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490615    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert\") pod \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") "
Sep 19 08:41:10.490806 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls\") pod \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\" (UID: \"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4\") "
Sep 19 08:41:10.490980 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.490949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" (UID: "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:41:10.491067 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.491039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" (UID: "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:41:10.491121 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.491072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" (UID: "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:41:10.492703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.492674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" (UID: "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:41:10.492820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.492740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55" (OuterVolumeSpecName: "kube-api-access-rqb55") pod "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" (UID: "5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4"). InnerVolumeSpecName "kube-api-access-rqb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:41:10.591863 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.591837    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:41:10.591863 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.591860    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:41:10.592004 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.591871    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:41:10.592004 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.591880    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb55\" (UniqueName: \"kubernetes.io/projected/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-kube-api-access-rqb55\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:41:10.592004 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.591889    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:41:10.636713 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.636689    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerID="20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3" exitCode=0
Sep 19 08:41:10.636857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.636750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerDied","Data":"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"}
Sep 19 08:41:10.636857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.636790    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"
Sep 19 08:41:10.636857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.636849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4" event={"ID":"5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4","Type":"ContainerDied","Data":"45bdf20b59a97e962bc10f3d6e3d1026ef8c4d0b0898632b2020ba761a8350eb"}
Sep 19 08:41:10.637023 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.636881    2566 scope.go:117] "RemoveContainer" containerID="20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"
Sep 19 08:41:10.645226 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.645205    2566 scope.go:117] "RemoveContainer" containerID="dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"
Sep 19 08:41:10.652043 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.652027    2566 scope.go:117] "RemoveContainer" containerID="5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"
Sep 19 08:41:10.659551 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.659524    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:41:10.659619 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.659593    2566 scope.go:117] "RemoveContainer" containerID="d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"
Sep 19 08:41:10.664286 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.664262    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-82hv4"]
Sep 19 08:41:10.666784 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.666750    2566 scope.go:117] "RemoveContainer" containerID="20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"
Sep 19 08:41:10.667035 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:41:10.667020    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3\": container with ID starting with 20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3 not found: ID does not exist" containerID="20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"
Sep 19 08:41:10.667119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667043    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3"} err="failed to get container status \"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3\": rpc error: code = NotFound desc = could not find container \"20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3\": container with ID starting with 20148f90f8a9b36a058cdc906e98a6ac9b8c69a5e0ecae47a2db11b09d9276d3 not found: ID does not exist"
Sep 19 08:41:10.667119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667059    2566 scope.go:117] "RemoveContainer" containerID="dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"
Sep 19 08:41:10.667318 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:41:10.667300    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11\": container with ID starting with dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11 not found: ID does not exist" containerID="dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"
Sep 19 08:41:10.667362 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667324    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11"} err="failed to get container status \"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11\": rpc error: code = NotFound desc = could not find container \"dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11\": container with ID starting with dcfbe8978be4c3910afe8b943d7081e576cdfe2f0e38b524d9a4368bbdb79a11 not found: ID does not exist"
Sep 19 08:41:10.667362 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667340    2566 scope.go:117] "RemoveContainer" containerID="5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"
Sep 19 08:41:10.667576 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:41:10.667559    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f\": container with ID starting with 5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f not found: ID does not exist" containerID="5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"
Sep 19 08:41:10.667639 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f"} err="failed to get container status \"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f\": rpc error: code = NotFound desc = could not find container \"5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f\": container with ID starting with 5b563863859e416e64579d336c78853fefe907bb97f96594084f3abaa26a186f not found: ID does not exist"
Sep 19 08:41:10.667639 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667604    2566 scope.go:117] "RemoveContainer" containerID="d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"
Sep 19 08:41:10.667869 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:41:10.667853    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0\": container with ID starting with d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0 not found: ID does not exist" containerID="d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"
Sep 19 08:41:10.667932 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:10.667877    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0"} err="failed to get container status \"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0\": rpc error: code = NotFound desc = could not find container \"d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0\": container with ID starting with d4bd302a8dc52f66517c9b0b2ce0c065b93b08025a0f8821f7ea0a33f0627ec0 not found: ID does not exist"
Sep 19 08:41:12.180243 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:41:12.180211    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" path="/var/lib/kubelet/pods/5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4/volumes"
Sep 19 08:42:15.858942 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:15.858905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:20.913596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.913558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:20.964176 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964147    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:42:20.964399 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964388    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964402    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964424    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964431    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent"
Sep 19 08:42:20.964446 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964443    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="storage-initializer"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964449    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="storage-initializer"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964461    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964466    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964474    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="storage-initializer"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964479    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="storage-initializer"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964491    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964497    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964504    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964509    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964548    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kserve-container"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964555    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964561    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="agent"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964567    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="836fe097-86aa-42af-b0df-68cc5062fb83" containerName="agent"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964571    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kube-rbac-proxy"
Sep 19 08:42:20.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.964577    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9d0542-d6ad-4e1d-b1fa-5bb5e496c5e4" containerName="kserve-container"
Sep 19 08:42:20.967447 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.967429    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:20.971511 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:42:20.971633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 19 08:42:20.971633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:42:20.971757 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971627    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:42:20.971850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971799    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:42:20.971850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:42:20.971946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.971827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 19 08:42:20.977830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:20.977804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:42:21.025480 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.025448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.025480 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.025481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.025664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.025506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.025664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.025589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.025664 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.025615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6td\" (UniqueName: \"kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.116037 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.115999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:21.125956 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.125931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126105 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.125962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6td\" (UniqueName: \"kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126105 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.125994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126105 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.126015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126105 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.126043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126504 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.126478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.126647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.126737 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.126664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.128393 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.128376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.133689 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.133661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6td\" (UniqueName: \"kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.278282 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.278193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:21.548092 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.548069    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:42:21.550354 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:42:21.550324    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2a1358_f925_4492_9fac_66885c8bbe83.slice/crio-2d234792b083115f02b74ab99515d0754e92d27baa9b06076950fd3101bfbae9 WatchSource:0}: Error finding container 2d234792b083115f02b74ab99515d0754e92d27baa9b06076950fd3101bfbae9: Status 404 returned error can't find the container with id 2d234792b083115f02b74ab99515d0754e92d27baa9b06076950fd3101bfbae9
Sep 19 08:42:21.899105 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.899071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerStarted","Data":"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"}
Sep 19 08:42:21.899299 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:21.899141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerStarted","Data":"2d234792b083115f02b74ab99515d0754e92d27baa9b06076950fd3101bfbae9"}
Sep 19 08:42:25.916229 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:25.916194    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerID="b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003" exitCode=0
Sep 19 08:42:25.916616 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:25.916266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerDied","Data":"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"}
Sep 19 08:42:25.917534 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:25.917520    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:42:45.730961 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:45.730916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:45.780939 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:45.780901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:47.022103 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:47.022007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerStarted","Data":"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"}
Sep 19 08:42:47.022103 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:47.022072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerStarted","Data":"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"}
Sep 19 08:42:47.022616 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:47.022402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:47.022616 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:47.022462    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:42:47.048920 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:47.048862    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podStartSLOduration=2.493710963 podStartE2EDuration="27.048842284s" totalImagesPullingTime="20.816207099s" totalInitContainerRuntime="3.738924222s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:42:47.047130058 +0000 UTC m=+1331.367801687" watchObservedRunningTime="2026-09-19 08:42:47.048842284 +0000 UTC m=+1331.369513872"
Sep 19 08:42:52.842099 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:52.842062    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:53.032725 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:42:53.032697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:43:15.983892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:15.983811    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:23.034659 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:23.034605    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:43:26.684976 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:26.684937    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:31.833026 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:31.832978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:32.378571 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:32.378537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:33.034315 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:33.034269    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:43:43.033763 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:43.033718    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:43:53.034317 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:43:53.034267    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:44:03.033693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:03.033649    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:44:13.548483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:13.548441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:44:21.033729 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.033666    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:44:21.034217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.034074    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" containerID="cri-o://d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79" gracePeriod=30
Sep 19 08:44:21.034217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.034119    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kube-rbac-proxy" containerID="cri-o://4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e" gracePeriod=30
Sep 19 08:44:21.059089 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.059047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:44:21.392722 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.392686    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerID="4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e" exitCode=2
Sep 19 08:44:21.392926 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:21.392757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerDied","Data":"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"}
Sep 19 08:44:23.026312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:23.026266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 19 08:44:23.034205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:23.034163    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 19 08:44:26.204511 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.204472    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:44:26.234077 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.234042    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:44:26.237606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.237589    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.240134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.240109    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 19 08:44:26.240247 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.240169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 19 08:44:26.247834 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.247810    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:44:26.362726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.362692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.362916 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.362734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.362916 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.362789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.362916 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.362822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqprn\" (UniqueName: \"kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.363035 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.362909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.447181 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.447150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:44:26.464155 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464347 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqprn\" (UniqueName: \"kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.464954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.465056 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.464995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.467131 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.467112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.472845 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.472819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqprn\" (UniqueName: \"kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.548113 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.548079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:26.696628 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.696602    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:44:26.766157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert\") pod \"0b2a1358-f925-4492-9fac-66885c8bbe83\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") "
Sep 19 08:44:26.766312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px6td\" (UniqueName: \"kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td\") pod \"0b2a1358-f925-4492-9fac-66885c8bbe83\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") "
Sep 19 08:44:26.766312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location\") pod \"0b2a1358-f925-4492-9fac-66885c8bbe83\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") "
Sep 19 08:44:26.766431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls\") pod \"0b2a1358-f925-4492-9fac-66885c8bbe83\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") "
Sep 19 08:44:26.766431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766372    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config\") pod \"0b2a1358-f925-4492-9fac-66885c8bbe83\" (UID: \"0b2a1358-f925-4492-9fac-66885c8bbe83\") "
Sep 19 08:44:26.766544 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b2a1358-f925-4492-9fac-66885c8bbe83" (UID: "0b2a1358-f925-4492-9fac-66885c8bbe83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:26.766609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b2a1358-f925-4492-9fac-66885c8bbe83" (UID: "0b2a1358-f925-4492-9fac-66885c8bbe83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:26.766665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766615    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:44:26.766826 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.766805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b2a1358-f925-4492-9fac-66885c8bbe83" (UID: "0b2a1358-f925-4492-9fac-66885c8bbe83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:26.768194 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.768174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td" (OuterVolumeSpecName: "kube-api-access-px6td") pod "0b2a1358-f925-4492-9fac-66885c8bbe83" (UID: "0b2a1358-f925-4492-9fac-66885c8bbe83"). InnerVolumeSpecName "kube-api-access-px6td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:44:26.768310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.768290    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b2a1358-f925-4492-9fac-66885c8bbe83" (UID: "0b2a1358-f925-4492-9fac-66885c8bbe83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:44:26.822602 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.822575    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:44:26.824375 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:44:26.824348    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61959e5_6491_4d16_bfc6_debdf1c00155.slice/crio-15c4b62c0f24f85c29cc42366896947b982d31af9f3756babc1cb59095bb958b WatchSource:0}: Error finding container 15c4b62c0f24f85c29cc42366896947b982d31af9f3756babc1cb59095bb958b: Status 404 returned error can't find the container with id 15c4b62c0f24f85c29cc42366896947b982d31af9f3756babc1cb59095bb958b
Sep 19 08:44:26.867719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.867689    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6td\" (UniqueName: \"kubernetes.io/projected/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-api-access-px6td\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:44:26.867719 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.867717    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b2a1358-f925-4492-9fac-66885c8bbe83-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:44:26.867897 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.867731    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b2a1358-f925-4492-9fac-66885c8bbe83-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:44:26.867897 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:26.867743    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b2a1358-f925-4492-9fac-66885c8bbe83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:44:27.418703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.418663    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerID="d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79" exitCode=0
Sep 19 08:44:27.419217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.418734    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"
Sep 19 08:44:27.419217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.418758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerDied","Data":"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"}
Sep 19 08:44:27.419217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.418848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f" event={"ID":"0b2a1358-f925-4492-9fac-66885c8bbe83","Type":"ContainerDied","Data":"2d234792b083115f02b74ab99515d0754e92d27baa9b06076950fd3101bfbae9"}
Sep 19 08:44:27.419217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.418886    2566 scope.go:117] "RemoveContainer" containerID="4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"
Sep 19 08:44:27.420629 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.420606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerStarted","Data":"84c55b63ed6e9d4787deb7c49e056cd412df47f6f1d7ce5673c6fae91184a7ee"}
Sep 19 08:44:27.420752 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.420672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerStarted","Data":"15c4b62c0f24f85c29cc42366896947b982d31af9f3756babc1cb59095bb958b"}
Sep 19 08:44:27.430014 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.429950    2566 scope.go:117] "RemoveContainer" containerID="d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"
Sep 19 08:44:27.438497 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.438467    2566 scope.go:117] "RemoveContainer" containerID="b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"
Sep 19 08:44:27.446937 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.446918    2566 scope.go:117] "RemoveContainer" containerID="4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"
Sep 19 08:44:27.447241 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:44:27.447221    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e\": container with ID starting with 4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e not found: ID does not exist" containerID="4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"
Sep 19 08:44:27.447320 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.447254    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e"} err="failed to get container status \"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e\": rpc error: code = NotFound desc = could not find container \"4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e\": container with ID starting with 4c0e7a565befac252aaff4908917722e4f574fc0385ddb4642321a83c476695e not found: ID does not exist"
Sep 19 08:44:27.447320 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.447281    2566 scope.go:117] "RemoveContainer" containerID="d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"
Sep 19 08:44:27.447575 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:44:27.447555    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79\": container with ID starting with d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79 not found: ID does not exist" containerID="d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"
Sep 19 08:44:27.447623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.447582    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79"} err="failed to get container status \"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79\": rpc error: code = NotFound desc = could not find container \"d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79\": container with ID starting with d8e4dd684158eac0bc52654e669334e7f576e58ef701ff4125fb03865e295c79 not found: ID does not exist"
Sep 19 08:44:27.447623 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.447598    2566 scope.go:117] "RemoveContainer" containerID="b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"
Sep 19 08:44:27.448098 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:44:27.448066    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003\": container with ID starting with b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003 not found: ID does not exist" containerID="b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"
Sep 19 08:44:27.448197 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.448103    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003"} err="failed to get container status \"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003\": rpc error: code = NotFound desc = could not find container \"b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003\": container with ID starting with b63470dd4171beb0942cb2ce17a423c90d6ddd367e6108ea4f6722f8ee061003 not found: ID does not exist"
Sep 19 08:44:27.458288 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.458258    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:44:27.463975 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:27.463938    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cz29f"]
Sep 19 08:44:28.180908 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:28.180832    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" path="/var/lib/kubelet/pods/0b2a1358-f925-4492-9fac-66885c8bbe83/volumes"
Sep 19 08:44:31.438040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:31.438006    2566 generic.go:356] "Generic (PLEG): container finished" podID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerID="84c55b63ed6e9d4787deb7c49e056cd412df47f6f1d7ce5673c6fae91184a7ee" exitCode=0
Sep 19 08:44:31.438559 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:31.438083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerDied","Data":"84c55b63ed6e9d4787deb7c49e056cd412df47f6f1d7ce5673c6fae91184a7ee"}
Sep 19 08:44:32.445146 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:32.445110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerStarted","Data":"201875533cd6adeed1b907f1c7971030492712ce8197b2a19b7ef8e4436d84e5"}
Sep 19 08:44:32.445602 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:32.445182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerStarted","Data":"7306905a65b20e58db9a1b1bf5f243fde954b9d64c09147a555a620c3c59692d"}
Sep 19 08:44:32.445602 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:32.445406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:32.468421 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:32.468372    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podStartSLOduration=2.342767813 podStartE2EDuration="6.468358447s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.125590634s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:32.467331303 +0000 UTC m=+1436.788002893" watchObservedRunningTime="2026-09-19 08:44:32.468358447 +0000 UTC m=+1436.789030035"
Sep 19 08:44:33.449284 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:33.449248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:44:39.458709 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:44:39.458677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:45:09.460158 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:09.460122    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 19 08:45:19.460123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:19.460080    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 19 08:45:29.460089 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:29.460042    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 19 08:45:38.392135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:38.392098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:45:39.459928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:39.459889    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 19 08:45:42.894305 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:42.894269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:45:43.164103 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:43.164015    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:45:49.951832 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:49.951797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:45:56.335695 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.335662    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:45:56.336189 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.335930    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" containerID="cri-o://7306905a65b20e58db9a1b1bf5f243fde954b9d64c09147a555a620c3c59692d" gracePeriod=30
Sep 19 08:45:56.336306 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.336225    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kube-rbac-proxy" containerID="cri-o://201875533cd6adeed1b907f1c7971030492712ce8197b2a19b7ef8e4436d84e5" gracePeriod=30
Sep 19 08:45:56.364467 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.364436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:45:56.759230 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.759194    2566 generic.go:356] "Generic (PLEG): container finished" podID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerID="201875533cd6adeed1b907f1c7971030492712ce8197b2a19b7ef8e4436d84e5" exitCode=2
Sep 19 08:45:56.759388 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:56.759257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerDied","Data":"201875533cd6adeed1b907f1c7971030492712ce8197b2a19b7ef8e4436d84e5"}
Sep 19 08:45:58.563915 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:58.563887    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:45:59.453717 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:59.453677    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 19 08:45:59.459882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:45:59.459859    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.24:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 19 08:46:01.503292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:01.503214    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:01.703522 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:01.703492    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:01.782713 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:01.782638    2566 generic.go:356] "Generic (PLEG): container finished" podID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerID="7306905a65b20e58db9a1b1bf5f243fde954b9d64c09147a555a620c3c59692d" exitCode=0
Sep 19 08:46:01.782848 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:01.782708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerDied","Data":"7306905a65b20e58db9a1b1bf5f243fde954b9d64c09147a555a620c3c59692d"}
Sep 19 08:46:01.873669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:01.873647    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:46:02.004150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert\") pod \"b61959e5-6491-4d16-bfc6-debdf1c00155\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") "
Sep 19 08:46:02.004150 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config\") pod \"b61959e5-6491-4d16-bfc6-debdf1c00155\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") "
Sep 19 08:46:02.004355 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls\") pod \"b61959e5-6491-4d16-bfc6-debdf1c00155\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") "
Sep 19 08:46:02.004355 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004249    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqprn\" (UniqueName: \"kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn\") pod \"b61959e5-6491-4d16-bfc6-debdf1c00155\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") "
Sep 19 08:46:02.004355 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location\") pod \"b61959e5-6491-4d16-bfc6-debdf1c00155\" (UID: \"b61959e5-6491-4d16-bfc6-debdf1c00155\") "
Sep 19 08:46:02.004507 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b61959e5-6491-4d16-bfc6-debdf1c00155" (UID: "b61959e5-6491-4d16-bfc6-debdf1c00155"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:02.004562 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004537    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b61959e5-6491-4d16-bfc6-debdf1c00155" (UID: "b61959e5-6491-4d16-bfc6-debdf1c00155"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:02.004657 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b61959e5-6491-4d16-bfc6-debdf1c00155" (UID: "b61959e5-6491-4d16-bfc6-debdf1c00155"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:02.004704 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004650    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:02.004704 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.004676    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61959e5-6491-4d16-bfc6-debdf1c00155-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:02.006243 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.006226    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b61959e5-6491-4d16-bfc6-debdf1c00155" (UID: "b61959e5-6491-4d16-bfc6-debdf1c00155"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:02.006308 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.006237    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn" (OuterVolumeSpecName: "kube-api-access-sqprn") pod "b61959e5-6491-4d16-bfc6-debdf1c00155" (UID: "b61959e5-6491-4d16-bfc6-debdf1c00155"). InnerVolumeSpecName "kube-api-access-sqprn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:02.105294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.105242    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b61959e5-6491-4d16-bfc6-debdf1c00155-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:02.105294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.105261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqprn\" (UniqueName: \"kubernetes.io/projected/b61959e5-6491-4d16-bfc6-debdf1c00155-kube-api-access-sqprn\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:02.105294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.105271    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61959e5-6491-4d16-bfc6-debdf1c00155-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:02.788472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.788443    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"
Sep 19 08:46:02.788472 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.788447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk" event={"ID":"b61959e5-6491-4d16-bfc6-debdf1c00155","Type":"ContainerDied","Data":"15c4b62c0f24f85c29cc42366896947b982d31af9f3756babc1cb59095bb958b"}
Sep 19 08:46:02.789013 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.788510    2566 scope.go:117] "RemoveContainer" containerID="201875533cd6adeed1b907f1c7971030492712ce8197b2a19b7ef8e4436d84e5"
Sep 19 08:46:02.795708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.795689    2566 scope.go:117] "RemoveContainer" containerID="7306905a65b20e58db9a1b1bf5f243fde954b9d64c09147a555a620c3c59692d"
Sep 19 08:46:02.802456 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.802439    2566 scope.go:117] "RemoveContainer" containerID="84c55b63ed6e9d4787deb7c49e056cd412df47f6f1d7ce5673c6fae91184a7ee"
Sep 19 08:46:02.809277 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.809255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:46:02.812509 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:02.812489    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rq8pk"]
Sep 19 08:46:03.572215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:03.572179    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:04.180360 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:04.180329    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" path="/var/lib/kubelet/pods/b61959e5-6491-4d16-bfc6-debdf1c00155/volumes"
Sep 19 08:46:19.046257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046223    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-39902e65"]
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="storage-initializer"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046586    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="storage-initializer"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046597    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kube-rbac-proxy"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046606    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kube-rbac-proxy"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046618    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="storage-initializer"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046626    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="storage-initializer"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046634    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046642    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046652    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046659    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container"
Sep 19 08:46:19.046670 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046668    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kube-rbac-proxy"
Sep 19 08:46:19.047208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046679    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kube-rbac-proxy"
Sep 19 08:46:19.047208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046752    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kserve-container"
Sep 19 08:46:19.047208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046782    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2a1358-f925-4492-9fac-66885c8bbe83" containerName="kube-rbac-proxy"
Sep 19 08:46:19.047208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046791    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kube-rbac-proxy"
Sep 19 08:46:19.047208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.046800    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61959e5-6491-4d16-bfc6-debdf1c00155" containerName="kserve-container"
Sep 19 08:46:19.049706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.049687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:19.052856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.052834    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:46:19.053001 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.052852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:46:19.053001 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.052878    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:46:19.053524 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.053502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-39902e65"]
Sep 19 08:46:19.114713 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.114676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfnl\" (UniqueName: \"kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl\") pod \"curl-metrics-39902e65\" (UID: \"6d8388cc-ac4a-443e-9d55-97edf3d4fad1\") " pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:19.215151 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.215098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfnl\" (UniqueName: \"kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl\") pod \"curl-metrics-39902e65\" (UID: \"6d8388cc-ac4a-443e-9d55-97edf3d4fad1\") " pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:19.223359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.223334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfnl\" (UniqueName: \"kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl\") pod \"curl-metrics-39902e65\" (UID: \"6d8388cc-ac4a-443e-9d55-97edf3d4fad1\") " pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:19.359573 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.359539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:19.622843 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.622815    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-39902e65"]
Sep 19 08:46:19.624963 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:46:19.624936    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8388cc_ac4a_443e_9d55_97edf3d4fad1.slice/crio-e5b16ec867cdd88169f805038545293ff252639a7cb06d0de242a511ce854e33 WatchSource:0}: Error finding container e5b16ec867cdd88169f805038545293ff252639a7cb06d0de242a511ce854e33: Status 404 returned error can't find the container with id e5b16ec867cdd88169f805038545293ff252639a7cb06d0de242a511ce854e33
Sep 19 08:46:19.852510 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:19.852480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-39902e65" event={"ID":"6d8388cc-ac4a-443e-9d55-97edf3d4fad1","Type":"ContainerStarted","Data":"e5b16ec867cdd88169f805038545293ff252639a7cb06d0de242a511ce854e33"}
Sep 19 08:46:20.857242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:20.857207    2566 generic.go:356] "Generic (PLEG): container finished" podID="6d8388cc-ac4a-443e-9d55-97edf3d4fad1" containerID="1f71690ce21d721e5b7d1b86fbb4bca4a8491b1c07ea8b01bbaeb989e617b4be" exitCode=0
Sep 19 08:46:20.857596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:20.857291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-39902e65" event={"ID":"6d8388cc-ac4a-443e-9d55-97edf3d4fad1","Type":"ContainerDied","Data":"1f71690ce21d721e5b7d1b86fbb4bca4a8491b1c07ea8b01bbaeb989e617b4be"}
Sep 19 08:46:21.980695 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:21.980672    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:21.999660 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:21.999639    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-39902e65_6d8388cc-ac4a-443e-9d55-97edf3d4fad1/curl-metrics-39902e65/0.log"
Sep 19 08:46:22.019461 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.019438    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-39902e65"]
Sep 19 08:46:22.023207 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.023184    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-39902e65"]
Sep 19 08:46:22.034276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.034256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgfnl\" (UniqueName: \"kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl\") pod \"6d8388cc-ac4a-443e-9d55-97edf3d4fad1\" (UID: \"6d8388cc-ac4a-443e-9d55-97edf3d4fad1\") "
Sep 19 08:46:22.036094 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.036071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl" (OuterVolumeSpecName: "kube-api-access-pgfnl") pod "6d8388cc-ac4a-443e-9d55-97edf3d4fad1" (UID: "6d8388cc-ac4a-443e-9d55-97edf3d4fad1"). InnerVolumeSpecName "kube-api-access-pgfnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:22.135033 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.134973    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgfnl\" (UniqueName: \"kubernetes.io/projected/6d8388cc-ac4a-443e-9d55-97edf3d4fad1-kube-api-access-pgfnl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.180104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.180072    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8388cc-ac4a-443e-9d55-97edf3d4fad1" path="/var/lib/kubelet/pods/6d8388cc-ac4a-443e-9d55-97edf3d4fad1/volumes"
Sep 19 08:46:22.865923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.865898    2566 scope.go:117] "RemoveContainer" containerID="1f71690ce21d721e5b7d1b86fbb4bca4a8491b1c07ea8b01bbaeb989e617b4be"
Sep 19 08:46:22.865923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:22.865923    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-39902e65"
Sep 19 08:46:27.404676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.404637    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-d0879e44"]
Sep 19 08:46:27.405156 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.404904    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8388cc-ac4a-443e-9d55-97edf3d4fad1" containerName="curl-metrics-39902e65"
Sep 19 08:46:27.405156 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.404916    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8388cc-ac4a-443e-9d55-97edf3d4fad1" containerName="curl-metrics-39902e65"
Sep 19 08:46:27.405156 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.404972    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8388cc-ac4a-443e-9d55-97edf3d4fad1" containerName="curl-metrics-39902e65"
Sep 19 08:46:27.408931 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.408910    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:27.411609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.411589    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:46:27.412867 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.412839    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:46:27.412985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.412876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:46:27.414294 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.414274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-d0879e44"]
Sep 19 08:46:27.469014 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.468991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zvm\" (UniqueName: \"kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm\") pod \"curl-metrics-d0879e44\" (UID: \"bd73eb9e-9195-44b2-be73-862f0032d709\") " pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:27.570184 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.570159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zvm\" (UniqueName: \"kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm\") pod \"curl-metrics-d0879e44\" (UID: \"bd73eb9e-9195-44b2-be73-862f0032d709\") " pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:27.578250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.578227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zvm\" (UniqueName: \"kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm\") pod \"curl-metrics-d0879e44\" (UID: \"bd73eb9e-9195-44b2-be73-862f0032d709\") " pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:27.719319 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.719248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:27.986923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:27.986876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-d0879e44"]
Sep 19 08:46:27.989498 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:46:27.989473    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd73eb9e_9195_44b2_be73_862f0032d709.slice/crio-46fadbdf83742d69ac4cfaa4e490281fdbee94b74d185e3397b88c8b00fec062 WatchSource:0}: Error finding container 46fadbdf83742d69ac4cfaa4e490281fdbee94b74d185e3397b88c8b00fec062: Status 404 returned error can't find the container with id 46fadbdf83742d69ac4cfaa4e490281fdbee94b74d185e3397b88c8b00fec062
Sep 19 08:46:28.891839 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:28.891805    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd73eb9e-9195-44b2-be73-862f0032d709" containerID="8d591e58bc3c7b861adae8f22823c68bd79488c1d1c10bee73ca63d6355379b1" exitCode=0
Sep 19 08:46:28.892244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:28.891853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-d0879e44" event={"ID":"bd73eb9e-9195-44b2-be73-862f0032d709","Type":"ContainerDied","Data":"8d591e58bc3c7b861adae8f22823c68bd79488c1d1c10bee73ca63d6355379b1"}
Sep 19 08:46:28.892244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:28.891879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-d0879e44" event={"ID":"bd73eb9e-9195-44b2-be73-862f0032d709","Type":"ContainerStarted","Data":"46fadbdf83742d69ac4cfaa4e490281fdbee94b74d185e3397b88c8b00fec062"}
Sep 19 08:46:30.009528 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.009504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:30.028907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.028884    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-d0879e44_bd73eb9e-9195-44b2-be73-862f0032d709/curl-metrics-d0879e44/0.log"
Sep 19 08:46:30.046104 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.046078    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-d0879e44"]
Sep 19 08:46:30.049606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.049585    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-d0879e44"]
Sep 19 08:46:30.088724 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.088702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47zvm\" (UniqueName: \"kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm\") pod \"bd73eb9e-9195-44b2-be73-862f0032d709\" (UID: \"bd73eb9e-9195-44b2-be73-862f0032d709\") "
Sep 19 08:46:30.090655 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.090632    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm" (OuterVolumeSpecName: "kube-api-access-47zvm") pod "bd73eb9e-9195-44b2-be73-862f0032d709" (UID: "bd73eb9e-9195-44b2-be73-862f0032d709"). InnerVolumeSpecName "kube-api-access-47zvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:30.180114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.180058    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd73eb9e-9195-44b2-be73-862f0032d709" path="/var/lib/kubelet/pods/bd73eb9e-9195-44b2-be73-862f0032d709/volumes"
Sep 19 08:46:30.189850 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.189827    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zvm\" (UniqueName: \"kubernetes.io/projected/bd73eb9e-9195-44b2-be73-862f0032d709-kube-api-access-47zvm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:46:30.899700 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.899668    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-d0879e44"
Sep 19 08:46:30.899901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:30.899681    2566 scope.go:117] "RemoveContainer" containerID="8d591e58bc3c7b861adae8f22823c68bd79488c1d1c10bee73ca63d6355379b1"
Sep 19 08:46:35.386906 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.386862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:35.414011 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.413982    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:46:35.414232 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.414221    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73eb9e-9195-44b2-be73-862f0032d709" containerName="curl-metrics-d0879e44"
Sep 19 08:46:35.414284 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.414234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73eb9e-9195-44b2-be73-862f0032d709" containerName="curl-metrics-d0879e44"
Sep 19 08:46:35.414329 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.414295    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd73eb9e-9195-44b2-be73-862f0032d709" containerName="curl-metrics-d0879e44"
Sep 19 08:46:35.418582 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.418564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.421138 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.421118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 19 08:46:35.421400 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.421381    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:46:35.421510 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.421493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:46:35.422152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.422132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:46:35.422368 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.422162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:46:35.422368 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.422162    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:46:35.422368 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.422161    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 19 08:46:35.427715 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.427693    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:46:35.521970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.521947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9b5b\" (UniqueName: \"kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.522110 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.521982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.522110 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.522009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.522213 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.522087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.522213 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.522150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.522447 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.522423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:35.623192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9b5b\" (UniqueName: \"kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623389 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623625 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.623970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.624017 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.623977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.625545 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.625530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.631782 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.631749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9b5b\" (UniqueName: \"kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b\") pod \"isvc-sklearn-predictor-868898bd6b-dbpdm\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.728561 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.728507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:35.996632 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:35.996610    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:46:35.998681 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:46:35.998656    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe7ab94_eac7_449a_994f_48e15abff7d0.slice/crio-d4d69f66ff3b867f6e76d8f3dce99db12adc66e214cb4d927a87eb30d7e837b9 WatchSource:0}: Error finding container d4d69f66ff3b867f6e76d8f3dce99db12adc66e214cb4d927a87eb30d7e837b9: Status 404 returned error can't find the container with id d4d69f66ff3b867f6e76d8f3dce99db12adc66e214cb4d927a87eb30d7e837b9
Sep 19 08:46:36.923006 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:36.922973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerStarted","Data":"db91ae803a11bcdb4282743bded2c70fb63f135e3c76afebc1303c70aeff1d16"}
Sep 19 08:46:36.923426 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:36.923038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerStarted","Data":"d4d69f66ff3b867f6e76d8f3dce99db12adc66e214cb4d927a87eb30d7e837b9"}
Sep 19 08:46:39.939321 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:39.939289    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerID="db91ae803a11bcdb4282743bded2c70fb63f135e3c76afebc1303c70aeff1d16" exitCode=0
Sep 19 08:46:39.939726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:39.939325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerDied","Data":"db91ae803a11bcdb4282743bded2c70fb63f135e3c76afebc1303c70aeff1d16"}
Sep 19 08:46:40.946269 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:40.946231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerStarted","Data":"a89cdc9a0bcc5eaadac483de8bbfaae473dc33b00fbee69b543f28d7d58c2b82"}
Sep 19 08:46:40.946744 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:40.946291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerStarted","Data":"232320addc59d568841a95634d66cf1244ca1f43f4c837380c4d70f6982ae643"}
Sep 19 08:46:40.946744 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:40.946454    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:40.965089 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:40.965049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podStartSLOduration=2.282999335 podStartE2EDuration="5.965036862s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682037527s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:46:40.963504153 +0000 UTC m=+1565.284175739" watchObservedRunningTime="2026-09-19 08:46:40.965036862 +0000 UTC m=+1565.285708448"
Sep 19 08:46:41.950325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:41.950289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:41.951668 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:41.951644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:46:42.956443 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:42.956399    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:46:47.960935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:47.960908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:46:47.961410 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:47.961384    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:46:57.961485 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:46:57.961439    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:07.962274 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:07.962234    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:16.358592 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:16.358547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:47:17.961944 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:17.961907    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:27.961693 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:27.961655    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:37.961359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:37.961318    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:47.962324 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:47.962284    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 19 08:47:49.428597 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:49.428557    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:47:54.632287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:54.632252    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:47:55.133397 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:55.133362    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:47:55.176946 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:47:55.176923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:48:07.889951 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:07.889919    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:09.899349 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:09.899318    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:11.909748 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:11.909715    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:13.920122 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:13.920089    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:15.930050 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:15.930017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:17.938566 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:17.938533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:19.948902 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:19.948866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:21.958625 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:21.958596    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:23.975140 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:23.975097    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:25.985283 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:25.985252    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:27.994603 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:27.994570    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:30.005680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:30.005647    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:32.016070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:32.016032    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:34.025954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:34.025920    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:36.038895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:36.038868    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:38.048981 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.048945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-dbpdm_bbe7ab94-eac7-449a-994f-48e15abff7d0/kserve-container/0.log"
Sep 19 08:48:38.161043 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.161012    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:48:38.161300 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.161260    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container" containerID="cri-o://232320addc59d568841a95634d66cf1244ca1f43f4c837380c4d70f6982ae643" gracePeriod=30
Sep 19 08:48:38.161450 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.161316    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kube-rbac-proxy" containerID="cri-o://a89cdc9a0bcc5eaadac483de8bbfaae473dc33b00fbee69b543f28d7d58c2b82" gracePeriod=30
Sep 19 08:48:38.181976 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.181950    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:48:38.371602 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.371577    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerID="a89cdc9a0bcc5eaadac483de8bbfaae473dc33b00fbee69b543f28d7d58c2b82" exitCode=2
Sep 19 08:48:38.371710 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:38.371621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerDied","Data":"a89cdc9a0bcc5eaadac483de8bbfaae473dc33b00fbee69b543f28d7d58c2b82"}
Sep 19 08:48:42.387963 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.387931    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerID="232320addc59d568841a95634d66cf1244ca1f43f4c837380c4d70f6982ae643" exitCode=0
Sep 19 08:48:42.388310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.388004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerDied","Data":"232320addc59d568841a95634d66cf1244ca1f43f4c837380c4d70f6982ae643"}
Sep 19 08:48:42.505240 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.505217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:48:42.640874 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.640846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9b5b\" (UniqueName: \"kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b\") pod \"bbe7ab94-eac7-449a-994f-48e15abff7d0\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") "
Sep 19 08:48:42.641054 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.640905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert\") pod \"bbe7ab94-eac7-449a-994f-48e15abff7d0\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") "
Sep 19 08:48:42.641054 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.640952    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config\") pod \"bbe7ab94-eac7-449a-994f-48e15abff7d0\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") "
Sep 19 08:48:42.641054 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.640979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls\") pod \"bbe7ab94-eac7-449a-994f-48e15abff7d0\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") "
Sep 19 08:48:42.641054 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.641002    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location\") pod \"bbe7ab94-eac7-449a-994f-48e15abff7d0\" (UID: \"bbe7ab94-eac7-449a-994f-48e15abff7d0\") "
Sep 19 08:48:42.641353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.641319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbe7ab94-eac7-449a-994f-48e15abff7d0" (UID: "bbe7ab94-eac7-449a-994f-48e15abff7d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:42.641353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.641342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbe7ab94-eac7-449a-994f-48e15abff7d0" (UID: "bbe7ab94-eac7-449a-994f-48e15abff7d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:42.641505 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.641349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbe7ab94-eac7-449a-994f-48e15abff7d0" (UID: "bbe7ab94-eac7-449a-994f-48e15abff7d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:42.642912 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.642895    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b" (OuterVolumeSpecName: "kube-api-access-c9b5b") pod "bbe7ab94-eac7-449a-994f-48e15abff7d0" (UID: "bbe7ab94-eac7-449a-994f-48e15abff7d0"). InnerVolumeSpecName "kube-api-access-c9b5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:48:42.643048 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.643036    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbe7ab94-eac7-449a-994f-48e15abff7d0" (UID: "bbe7ab94-eac7-449a-994f-48e15abff7d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:48:42.741928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.741900    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9b5b\" (UniqueName: \"kubernetes.io/projected/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-api-access-c9b5b\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:48:42.741928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.741928    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:48:42.742114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.741941    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe7ab94-eac7-449a-994f-48e15abff7d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:48:42.742114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.741954    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe7ab94-eac7-449a-994f-48e15abff7d0-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:48:42.742114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:42.741966    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe7ab94-eac7-449a-994f-48e15abff7d0-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:48:43.340414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.340377    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:48:43.368898 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.368871    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:48:43.369139 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369129    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container"
Sep 19 08:48:43.369177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369142    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container"
Sep 19 08:48:43.369177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369156    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kube-rbac-proxy"
Sep 19 08:48:43.369177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369162    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kube-rbac-proxy"
Sep 19 08:48:43.369177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369168    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="storage-initializer"
Sep 19 08:48:43.369177 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369173    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="storage-initializer"
Sep 19 08:48:43.369325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369217    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kserve-container"
Sep 19 08:48:43.369325 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.369224    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" containerName="kube-rbac-proxy"
Sep 19 08:48:43.372299 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.372283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.374300 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.374274    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 19 08:48:43.374429 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.374407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 19 08:48:43.382002 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.381980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:48:43.393346 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.393327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm" event={"ID":"bbe7ab94-eac7-449a-994f-48e15abff7d0","Type":"ContainerDied","Data":"d4d69f66ff3b867f6e76d8f3dce99db12adc66e214cb4d927a87eb30d7e837b9"}
Sep 19 08:48:43.393680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.393381    2566 scope.go:117] "RemoveContainer" containerID="a89cdc9a0bcc5eaadac483de8bbfaae473dc33b00fbee69b543f28d7d58c2b82"
Sep 19 08:48:43.393680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.393335    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"
Sep 19 08:48:43.401492 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.401437    2566 scope.go:117] "RemoveContainer" containerID="232320addc59d568841a95634d66cf1244ca1f43f4c837380c4d70f6982ae643"
Sep 19 08:48:43.408613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.408598    2566 scope.go:117] "RemoveContainer" containerID="db91ae803a11bcdb4282743bded2c70fb63f135e3c76afebc1303c70aeff1d16"
Sep 19 08:48:43.415106 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.415087    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:48:43.416732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.416715    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-dbpdm"]
Sep 19 08:48:43.446753 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.446735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.446842 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.446822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rml4\" (UniqueName: \"kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.446891 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.446863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.446941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.446898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.446994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.446979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.547706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.547681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rml4\" (UniqueName: \"kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.547869 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.547722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.547869 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.547757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.547869 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.547824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.547869 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.547862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.548267 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.548244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.548461 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.548443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.548517 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.548446    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.550112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.550097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.555531 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.555510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:48:43.555718 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.555697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rml4\" (UniqueName: \"kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.682193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.682166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:48:43.952819 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:43.952791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:48:43.954728 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:48:43.954703    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273baf1d_34e7_4dbb_a2c2_3f97b37cc94e.slice/crio-632394bfefb8ead26f47bafeb40ee7a52f7907b12d4f97f7823ed53ac94f7160 WatchSource:0}: Error finding container 632394bfefb8ead26f47bafeb40ee7a52f7907b12d4f97f7823ed53ac94f7160: Status 404 returned error can't find the container with id 632394bfefb8ead26f47bafeb40ee7a52f7907b12d4f97f7823ed53ac94f7160
Sep 19 08:48:44.181134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:44.181104    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe7ab94-eac7-449a-994f-48e15abff7d0" path="/var/lib/kubelet/pods/bbe7ab94-eac7-449a-994f-48e15abff7d0/volumes"
Sep 19 08:48:44.399314 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:44.399276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerStarted","Data":"aa78636b38f45ef110f1931bebd0591c5096d8fb9e846efb451656d22757edab"}
Sep 19 08:48:44.399733 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:44.399329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerStarted","Data":"632394bfefb8ead26f47bafeb40ee7a52f7907b12d4f97f7823ed53ac94f7160"}
Sep 19 08:48:48.414371 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:48.414334    2566 generic.go:356] "Generic (PLEG): container finished" podID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerID="aa78636b38f45ef110f1931bebd0591c5096d8fb9e846efb451656d22757edab" exitCode=0
Sep 19 08:48:48.414803 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:48.414381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerDied","Data":"aa78636b38f45ef110f1931bebd0591c5096d8fb9e846efb451656d22757edab"}
Sep 19 08:48:48.415504 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:48:48.415488    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:49:26.446747 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:49:26.446684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:00.843706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:00.843658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:05.980484 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:05.980434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:06.133031 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:06.132985    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:42.920525 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:42.920499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerStarted","Data":"86e777755ec44f979e8c2a386996d58b3d33b21d77adf126840c239b0b0ba805"}
Sep 19 08:50:43.925819 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:43.925779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerStarted","Data":"7813dd93785319b4b66e6bb1c31895bfd6497f05f8a11196c6982fb9baac0424"}
Sep 19 08:50:43.926180 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:43.925925    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:50:43.948922 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:43.948876    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" podStartSLOduration=2.755876823 podStartE2EDuration="2m0.94886227s" totalImagesPullingTime="1m54.34044568s" totalInitContainerRuntime="3.852539767s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:50:43.946686358 +0000 UTC m=+1808.267357945" watchObservedRunningTime="2026-09-19 08:50:43.94886227 +0000 UTC m=+1808.269533854"
Sep 19 08:50:44.929621 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:44.929587    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:50:48.408718 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:48.408686    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:50.938099 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:50.938072    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:50:53.553788 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:53.553735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:50:53.801887 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:50:53.801836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:04.955989 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:04.955953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:05.055359 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:05.055326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:20.943529 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:20.943483    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 19 08:51:28.937265 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:28.937222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:30.940888 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:30.940859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:51:34.077714 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.077672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:34.112904 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.112879    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:51:34.115407 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.115391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.118051 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.118023    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 19 08:51:34.118051 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.118048    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 19 08:51:34.126752 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.126726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:51:34.149086 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.149057    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:51:34.151483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.151467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.153540 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.153515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:34.153739 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.153722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 19 08:51:34.162607 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.162586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:51:34.239397 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.239397 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqhk\" (UniqueName: \"kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.239606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.239856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wnmv\" (UniqueName: \"kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.239856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.239629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.340654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.340654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqhk\" (UniqueName: \"kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.340901 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wnmv\" (UniqueName: \"kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.341297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.340906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.341297 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:51:34.340945    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:51:34.341297 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:51:34.341034    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls podName:348c2996-ba4b-41ba-ae7d-46f5a0751154 nodeName:}" failed. No retries permitted until 2026-09-19 08:51:34.841002324 +0000 UTC m=+1859.161673894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-nzq9g" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:51:34.341297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.341510 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.341510 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.341607 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.341607 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.341607 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.341585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.343362 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.343343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.351386 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.351363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wnmv\" (UniqueName: \"kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.351469 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.351442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqhk\" (UniqueName: \"kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk\") pod \"isvc-canary-create-v2-predictor-667f95d746-qmv8p\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.367330 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.367291    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:34.425303 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.425274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:34.468539 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.468504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:34.696232 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.696209    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:51:34.697591 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:51:34.697568    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289fbc40_0793_4dae_9978_ab2b8aae6f10.slice/crio-3170a8e7c3e774ac6301525e9d30f0e3c9fa81c03581566bf7730e3d1275b492 WatchSource:0}: Error finding container 3170a8e7c3e774ac6301525e9d30f0e3c9fa81c03581566bf7730e3d1275b492: Status 404 returned error can't find the container with id 3170a8e7c3e774ac6301525e9d30f0e3c9fa81c03581566bf7730e3d1275b492
Sep 19 08:51:34.845217 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.845171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:34.847690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:34.847661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-nzq9g\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:35.061047 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:35.060964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:35.116660 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:35.116084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerStarted","Data":"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"}
Sep 19 08:51:35.116660 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:35.116163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerStarted","Data":"3170a8e7c3e774ac6301525e9d30f0e3c9fa81c03581566bf7730e3d1275b492"}
Sep 19 08:51:35.334252 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:35.334176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:51:35.336646 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:51:35.336614    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348c2996_ba4b_41ba_ae7d_46f5a0751154.slice/crio-2099a4b6bf5246f856817a0e7118dfefd4727177fa5f1d71e1226f08d8c796ae WatchSource:0}: Error finding container 2099a4b6bf5246f856817a0e7118dfefd4727177fa5f1d71e1226f08d8c796ae: Status 404 returned error can't find the container with id 2099a4b6bf5246f856817a0e7118dfefd4727177fa5f1d71e1226f08d8c796ae
Sep 19 08:51:36.121070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:36.121034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerStarted","Data":"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"}
Sep 19 08:51:36.121495 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:36.121094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerStarted","Data":"2099a4b6bf5246f856817a0e7118dfefd4727177fa5f1d71e1226f08d8c796ae"}
Sep 19 08:51:37.473259 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:37.473226    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:51:37.473661 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:37.473451    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kserve-container" containerID="cri-o://86e777755ec44f979e8c2a386996d58b3d33b21d77adf126840c239b0b0ba805" gracePeriod=30
Sep 19 08:51:37.473661 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:37.473507    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kube-rbac-proxy" containerID="cri-o://7813dd93785319b4b66e6bb1c31895bfd6497f05f8a11196c6982fb9baac0424" gracePeriod=30
Sep 19 08:51:37.494462 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:37.494424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:38.130402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:38.130368    2566 generic.go:356] "Generic (PLEG): container finished" podID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerID="7813dd93785319b4b66e6bb1c31895bfd6497f05f8a11196c6982fb9baac0424" exitCode=2
Sep 19 08:51:38.130572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:38.130437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerDied","Data":"7813dd93785319b4b66e6bb1c31895bfd6497f05f8a11196c6982fb9baac0424"}
Sep 19 08:51:39.135669 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:39.135638    2566 generic.go:356] "Generic (PLEG): container finished" podID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerID="433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5" exitCode=0
Sep 19 08:51:39.136046 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:39.135710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerDied","Data":"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"}
Sep 19 08:51:40.140910 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.140876    2566 generic.go:356] "Generic (PLEG): container finished" podID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerID="2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6" exitCode=0
Sep 19 08:51:40.141275 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.140949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerDied","Data":"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"}
Sep 19 08:51:40.143310 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.143291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerStarted","Data":"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"}
Sep 19 08:51:40.143422 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.143350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerStarted","Data":"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"}
Sep 19 08:51:40.143565 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.143550    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:40.143640 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.143592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:40.145138 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.145114    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:51:40.186411 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.186351    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podStartSLOduration=2.280876821 podStartE2EDuration="6.186334921s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.9054581s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:40.183619422 +0000 UTC m=+1864.504291010" watchObservedRunningTime="2026-09-19 08:51:40.186334921 +0000 UTC m=+1864.507006531"
Sep 19 08:51:40.934406 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:40.934360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 19 08:51:41.150878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:41.150836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerStarted","Data":"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"}
Sep 19 08:51:41.151323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:41.150922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerStarted","Data":"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"}
Sep 19 08:51:41.151323 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:41.151289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:41.151475 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:41.151444    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:51:41.169931 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:41.169889    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podStartSLOduration=3.228072584 podStartE2EDuration="7.169875859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941803275s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:41.168221994 +0000 UTC m=+1865.488893582" watchObservedRunningTime="2026-09-19 08:51:41.169875859 +0000 UTC m=+1865.490547446"
Sep 19 08:51:42.155487 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.155455    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:42.156717 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.156688    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:51:42.649635 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.649599    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:42.680999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.680968    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:51:42.683257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.683243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.690056 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.690039    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 19 08:51:42.690143 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.690043    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 19 08:51:42.694672 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.694647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:51:42.812436 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.812400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.812613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.812445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.812613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.812467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.812613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.812564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.812720 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.812635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrsl\" (UniqueName: \"kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.913978 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.913878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.913978 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.913940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914173 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:51:42.914101    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 19 08:51:42.914173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914173 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914287 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:51:42.914190    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls podName:bd3894eb-a0b4-4968-9bdf-3faff7664aeb nodeName:}" failed. No retries permitted until 2026-09-19 08:51:43.414173863 +0000 UTC m=+1867.734845432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls") pod "isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 19 08:51:42.914287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrsl\" (UniqueName: \"kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914917 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.914963 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.914897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.924614 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.924597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrsl\" (UniqueName: \"kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:42.989889 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:42.989858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:51:43.158998 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:43.158956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:51:43.418950 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:43.418920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:43.421110 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:43.421089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:43.592725 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:43.592692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:43.861534 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:43.861505    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:51:43.863163 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:51:43.863140    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3894eb_a0b4_4968_9bdf_3faff7664aeb.slice/crio-3d0730289cfe775cb839b08d1d013d071d387523e80b4faa5d9ea5d3d5e875ed WatchSource:0}: Error finding container 3d0730289cfe775cb839b08d1d013d071d387523e80b4faa5d9ea5d3d5e875ed: Status 404 returned error can't find the container with id 3d0730289cfe775cb839b08d1d013d071d387523e80b4faa5d9ea5d3d5e875ed
Sep 19 08:51:44.165067 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:44.164980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerStarted","Data":"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"}
Sep 19 08:51:44.165067 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:44.165050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerStarted","Data":"3d0730289cfe775cb839b08d1d013d071d387523e80b4faa5d9ea5d3d5e875ed"}
Sep 19 08:51:45.170756 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.170726    2566 generic.go:356] "Generic (PLEG): container finished" podID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerID="86e777755ec44f979e8c2a386996d58b3d33b21d77adf126840c239b0b0ba805" exitCode=0
Sep 19 08:51:45.171189 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.170805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerDied","Data":"86e777755ec44f979e8c2a386996d58b3d33b21d77adf126840c239b0b0ba805"}
Sep 19 08:51:45.204881 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.204857    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:51:45.336596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rml4\" (UniqueName: \"kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4\") pod \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") "
Sep 19 08:51:45.336596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location\") pod \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") "
Sep 19 08:51:45.336596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls\") pod \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") "
Sep 19 08:51:45.336892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336636    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config\") pod \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") "
Sep 19 08:51:45.336892 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert\") pod \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\" (UID: \"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e\") "
Sep 19 08:51:45.337008 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.336977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" (UID: "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:45.337063 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.337005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" (UID: "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:51:45.337063 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.337039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" (UID: "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:51:45.337208 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.337186    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:51:45.337276 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.337215    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:51:45.338730 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.338704    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" (UID: "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:51:45.338841 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.338727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4" (OuterVolumeSpecName: "kube-api-access-7rml4") pod "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" (UID: "273baf1d-34e7-4dbb-a2c2-3f97b37cc94e"). InnerVolumeSpecName "kube-api-access-7rml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:51:45.438108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.438073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rml4\" (UniqueName: \"kubernetes.io/projected/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-kube-api-access-7rml4\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:51:45.438108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.438102    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:51:45.438108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:45.438112    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:51:46.156234 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.156204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:51:46.156754 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.156728    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:51:46.178911 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.178889    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"
Sep 19 08:51:46.180947 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.180920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx" event={"ID":"273baf1d-34e7-4dbb-a2c2-3f97b37cc94e","Type":"ContainerDied","Data":"632394bfefb8ead26f47bafeb40ee7a52f7907b12d4f97f7823ed53ac94f7160"}
Sep 19 08:51:46.181066 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.181029    2566 scope.go:117] "RemoveContainer" containerID="7813dd93785319b4b66e6bb1c31895bfd6497f05f8a11196c6982fb9baac0424"
Sep 19 08:51:46.195608 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.195590    2566 scope.go:117] "RemoveContainer" containerID="86e777755ec44f979e8c2a386996d58b3d33b21d77adf126840c239b0b0ba805"
Sep 19 08:51:46.202705 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.202687    2566 scope.go:117] "RemoveContainer" containerID="aa78636b38f45ef110f1931bebd0591c5096d8fb9e846efb451656d22757edab"
Sep 19 08:51:46.205793 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.205756    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:51:46.210897 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:46.210880    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-h4dhx"]
Sep 19 08:51:48.165418 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:48.165389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:51:48.166072 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:48.166045    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:51:48.181137 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:48.181111    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" path="/var/lib/kubelet/pods/273baf1d-34e7-4dbb-a2c2-3f97b37cc94e/volumes"
Sep 19 08:51:50.196160 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:50.196083    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerID="2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09" exitCode=0
Sep 19 08:51:50.196160 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:50.196124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerDied","Data":"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"}
Sep 19 08:51:51.202485 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.202454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerStarted","Data":"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"}
Sep 19 08:51:51.202907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.202518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerStarted","Data":"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"}
Sep 19 08:51:51.202907 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.202801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:51.203020 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.202976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:51.204220 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.204179    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 19 08:51:51.222212 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:51.222160    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podStartSLOduration=3.220598136 podStartE2EDuration="9.222146509s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.001548373s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:51.221422916 +0000 UTC m=+1875.542094505" watchObservedRunningTime="2026-09-19 08:51:51.222146509 +0000 UTC m=+1875.542818092"
Sep 19 08:51:52.207257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:52.207209    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 19 08:51:56.157468 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:56.157433    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:51:57.212512 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:57.212484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:51:57.212993 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:57.212969    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 19 08:51:58.167032 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:51:58.166950    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:06.156786 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:06.156732    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:52:07.213641 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:07.213613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:52:08.166441 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:08.166403    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:14.614498 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:14.614467    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:52:14.614970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:14.614800    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container" containerID="cri-o://2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd" gracePeriod=30
Sep 19 08:52:14.614970 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:14.614814    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kube-rbac-proxy" containerID="cri-o://eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109" gracePeriod=30
Sep 19 08:52:14.642344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:14.642318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:52:15.313868 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.313822    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerID="eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109" exitCode=2
Sep 19 08:52:15.313982 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.313894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerDied","Data":"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"}
Sep 19 08:52:15.440244 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.440224    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:52:15.494071 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494047    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdrsl\" (UniqueName: \"kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl\") pod \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") "
Sep 19 08:52:15.494192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert\") pod \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") "
Sep 19 08:52:15.494192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config\") pod \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") "
Sep 19 08:52:15.494192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") pod \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") "
Sep 19 08:52:15.494192 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location\") pod \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\" (UID: \"bd3894eb-a0b4-4968-9bdf-3faff7664aeb\") "
Sep 19 08:52:15.494531 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd3894eb-a0b4-4968-9bdf-3faff7664aeb" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:52:15.494602 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.494537    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd3894eb-a0b4-4968-9bdf-3faff7664aeb" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:52:15.496119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.496099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl" (OuterVolumeSpecName: "kube-api-access-gdrsl") pod "bd3894eb-a0b4-4968-9bdf-3faff7664aeb" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb"). InnerVolumeSpecName "kube-api-access-gdrsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:52:15.496119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.496098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd3894eb-a0b4-4968-9bdf-3faff7664aeb" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:52:15.520204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.520145    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd3894eb-a0b4-4968-9bdf-3faff7664aeb" (UID: "bd3894eb-a0b4-4968-9bdf-3faff7664aeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:52:15.595588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.595562    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:52:15.595588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.595585    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:52:15.595726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.595595    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:52:15.595726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.595604    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdrsl\" (UniqueName: \"kubernetes.io/projected/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-kube-api-access-gdrsl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:52:15.595726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:15.595613    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3894eb-a0b4-4968-9bdf-3faff7664aeb-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:52:16.157551 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.157514    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:52:16.319824 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.319791    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerID="2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd" exitCode=0
Sep 19 08:52:16.319985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.319856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerDied","Data":"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"}
Sep 19 08:52:16.319985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.319862    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"
Sep 19 08:52:16.319985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.319903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9" event={"ID":"bd3894eb-a0b4-4968-9bdf-3faff7664aeb","Type":"ContainerDied","Data":"3d0730289cfe775cb839b08d1d013d071d387523e80b4faa5d9ea5d3d5e875ed"}
Sep 19 08:52:16.319985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.319928    2566 scope.go:117] "RemoveContainer" containerID="eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"
Sep 19 08:52:16.327656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.327642    2566 scope.go:117] "RemoveContainer" containerID="2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"
Sep 19 08:52:16.334402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.334382    2566 scope.go:117] "RemoveContainer" containerID="2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"
Sep 19 08:52:16.338047 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.338024    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:52:16.341378 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341359    2566 scope.go:117] "RemoveContainer" containerID="eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"
Sep 19 08:52:16.341563 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341546    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-8tsh9"]
Sep 19 08:52:16.341637 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:52:16.341619    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109\": container with ID starting with eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109 not found: ID does not exist" containerID="eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"
Sep 19 08:52:16.341677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341646    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109"} err="failed to get container status \"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109\": rpc error: code = NotFound desc = could not find container \"eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109\": container with ID starting with eb7bce4bc9e3bd4e35e6ebccf2f5cfbc632b936f52b6ac03025d343c9f335109 not found: ID does not exist"
Sep 19 08:52:16.341677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341664    2566 scope.go:117] "RemoveContainer" containerID="2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"
Sep 19 08:52:16.341921 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:52:16.341901    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd\": container with ID starting with 2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd not found: ID does not exist" containerID="2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"
Sep 19 08:52:16.342006 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd"} err="failed to get container status \"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd\": rpc error: code = NotFound desc = could not find container \"2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd\": container with ID starting with 2e891044cf29851f7666a62635b5f78e208879d814e45320c1ad0b59a89cd7bd not found: ID does not exist"
Sep 19 08:52:16.342006 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.341938    2566 scope.go:117] "RemoveContainer" containerID="2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"
Sep 19 08:52:16.342196 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:52:16.342178    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09\": container with ID starting with 2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09 not found: ID does not exist" containerID="2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"
Sep 19 08:52:16.342249 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:16.342205    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09"} err="failed to get container status \"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09\": rpc error: code = NotFound desc = could not find container \"2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09\": container with ID starting with 2edeba5e1a866f04758a65baa9ca1bc9178dc2512fd3ab88abf4304f07638c09 not found: ID does not exist"
Sep 19 08:52:18.166413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:18.166374    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:18.180655 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:18.180625    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" path="/var/lib/kubelet/pods/bd3894eb-a0b4-4968-9bdf-3faff7664aeb/volumes"
Sep 19 08:52:19.760977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.760942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:52:19.800494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800459    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:52:19.800734 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800722    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kserve-container"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800735    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kserve-container"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800743    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kube-rbac-proxy"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800750    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kube-rbac-proxy"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="storage-initializer"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800763    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="storage-initializer"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800783    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kube-rbac-proxy"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800789    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kube-rbac-proxy"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800795    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="storage-initializer"
Sep 19 08:52:19.800802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800800    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="storage-initializer"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800818    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800824    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800866    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kserve-container"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800875    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="273baf1d-34e7-4dbb-a2c2-3f97b37cc94e" containerName="kube-rbac-proxy"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800881    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kserve-container"
Sep 19 08:52:19.801068 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.800886    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3894eb-a0b4-4968-9bdf-3faff7664aeb" containerName="kube-rbac-proxy"
Sep 19 08:52:19.803483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.803467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.805802 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.805780    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 19 08:52:19.805928 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.805822    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 19 08:52:19.812290 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.812272    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:52:19.928569 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.928537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.928708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.928582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.928708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.928613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgzl\" (UniqueName: \"kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.928708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.928632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.928708 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.928661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:19.992119 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:19.992084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:52:20.029673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.029620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.029673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.029655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgzl\" (UniqueName: \"kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.029857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.029675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.029857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.029703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.029857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.029723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.030206 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.030183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.030460 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.030442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.030504 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.030458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.032065 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.032050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.037856 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.037837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgzl\" (UniqueName: \"kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.114095 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.114068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:20.381272 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:20.381207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:52:20.383741 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:52:20.383713    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d47f839_9e02_4747_a0ac_840279bf5bca.slice/crio-60a615fe15b7e5789f363f85891651edadf53bd03ba9a103a85fece85c8cbfa9 WatchSource:0}: Error finding container 60a615fe15b7e5789f363f85891651edadf53bd03ba9a103a85fece85c8cbfa9: Status 404 returned error can't find the container with id 60a615fe15b7e5789f363f85891651edadf53bd03ba9a103a85fece85c8cbfa9
Sep 19 08:52:21.341151 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:21.341117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerStarted","Data":"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"}
Sep 19 08:52:21.341525 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:21.341177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerStarted","Data":"60a615fe15b7e5789f363f85891651edadf53bd03ba9a103a85fece85c8cbfa9"}
Sep 19 08:52:24.357303 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:24.357273    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerID="4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757" exitCode=0
Sep 19 08:52:24.357619 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:24.357349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerDied","Data":"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"}
Sep 19 08:52:25.364029 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:25.363992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerStarted","Data":"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"}
Sep 19 08:52:25.364391 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:25.364058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerStarted","Data":"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"}
Sep 19 08:52:25.364391 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:25.364252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:25.383460 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:25.383418    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" podStartSLOduration=2.5012738260000003 podStartE2EDuration="6.383404856s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88213103s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:52:25.38220322 +0000 UTC m=+1909.702874806" watchObservedRunningTime="2026-09-19 08:52:25.383404856 +0000 UTC m=+1909.704076443"
Sep 19 08:52:26.157097 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:26.157056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:52:26.368586 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:26.368559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:28.166349 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:28.166308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:32.377423 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:32.377394    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:52:36.157248 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:36.157212    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:52:38.166108 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:38.166070    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:46.157340 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:46.157300    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:52:48.166754 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:48.166714    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:54.176463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:54.176419    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 19 08:52:56.157977 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.157948    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:52:56.271621 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.271589    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:52:56.271832 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.271808    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container" containerID="cri-o://59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c" gracePeriod=30
Sep 19 08:52:56.271911 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.271861    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kube-rbac-proxy" containerID="cri-o://9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6" gracePeriod=30
Sep 19 08:52:56.272653 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.272636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:52:56.487728 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.487657    2566 generic.go:356] "Generic (PLEG): container finished" podID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerID="9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6" exitCode=2
Sep 19 08:52:56.487870 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:56.487731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerDied","Data":"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"}
Sep 19 08:52:58.159883 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:52:58.159847    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 19 08:53:00.813633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.813612    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:53:00.856825 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.856792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wnmv\" (UniqueName: \"kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv\") pod \"348c2996-ba4b-41ba-ae7d-46f5a0751154\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") "
Sep 19 08:53:00.856999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.856843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location\") pod \"348c2996-ba4b-41ba-ae7d-46f5a0751154\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") "
Sep 19 08:53:00.856999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.856885    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert\") pod \"348c2996-ba4b-41ba-ae7d-46f5a0751154\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") "
Sep 19 08:53:00.856999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.856925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") pod \"348c2996-ba4b-41ba-ae7d-46f5a0751154\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") "
Sep 19 08:53:00.856999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.856975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config\") pod \"348c2996-ba4b-41ba-ae7d-46f5a0751154\" (UID: \"348c2996-ba4b-41ba-ae7d-46f5a0751154\") "
Sep 19 08:53:00.857293 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.857220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "348c2996-ba4b-41ba-ae7d-46f5a0751154" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:00.857414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.857310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "348c2996-ba4b-41ba-ae7d-46f5a0751154" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:00.857414 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.857375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "348c2996-ba4b-41ba-ae7d-46f5a0751154" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:00.858987 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.858961    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "348c2996-ba4b-41ba-ae7d-46f5a0751154" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:00.858987 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.858981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv" (OuterVolumeSpecName: "kube-api-access-8wnmv") pod "348c2996-ba4b-41ba-ae7d-46f5a0751154" (UID: "348c2996-ba4b-41ba-ae7d-46f5a0751154"). InnerVolumeSpecName "kube-api-access-8wnmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:00.958250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.958177    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348c2996-ba4b-41ba-ae7d-46f5a0751154-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:00.958250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.958204    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:00.958250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.958213    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348c2996-ba4b-41ba-ae7d-46f5a0751154-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:00.958250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.958221    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:00.958250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:00.958230    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wnmv\" (UniqueName: \"kubernetes.io/projected/348c2996-ba4b-41ba-ae7d-46f5a0751154-kube-api-access-8wnmv\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:01.507989 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.507954    2566 generic.go:356] "Generic (PLEG): container finished" podID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerID="59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c" exitCode=0
Sep 19 08:53:01.508152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.508008    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"
Sep 19 08:53:01.508152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.508014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerDied","Data":"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"}
Sep 19 08:53:01.508152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.508059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g" event={"ID":"348c2996-ba4b-41ba-ae7d-46f5a0751154","Type":"ContainerDied","Data":"2099a4b6bf5246f856817a0e7118dfefd4727177fa5f1d71e1226f08d8c796ae"}
Sep 19 08:53:01.508152 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.508084    2566 scope.go:117] "RemoveContainer" containerID="9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"
Sep 19 08:53:01.516492 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.516478    2566 scope.go:117] "RemoveContainer" containerID="59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"
Sep 19 08:53:01.523187 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.523171    2566 scope.go:117] "RemoveContainer" containerID="2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"
Sep 19 08:53:01.529382 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.529360    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:53:01.530102 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530083    2566 scope.go:117] "RemoveContainer" containerID="9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"
Sep 19 08:53:01.530369 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:01.530353    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6\": container with ID starting with 9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6 not found: ID does not exist" containerID="9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"
Sep 19 08:53:01.530425 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530375    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6"} err="failed to get container status \"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6\": rpc error: code = NotFound desc = could not find container \"9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6\": container with ID starting with 9d0b92b85ea61ff63cb2637d810f5dc91dcbb6834519cdd409313c454e1160c6 not found: ID does not exist"
Sep 19 08:53:01.530425 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530390    2566 scope.go:117] "RemoveContainer" containerID="59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"
Sep 19 08:53:01.530605 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:01.530590    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c\": container with ID starting with 59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c not found: ID does not exist" containerID="59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"
Sep 19 08:53:01.530644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530609    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c"} err="failed to get container status \"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c\": rpc error: code = NotFound desc = could not find container \"59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c\": container with ID starting with 59939bab492bc333099dee46ef05e52a7ba6159c30356a750bb7826a51385c7c not found: ID does not exist"
Sep 19 08:53:01.530644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530621    2566 scope.go:117] "RemoveContainer" containerID="2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"
Sep 19 08:53:01.530880 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:01.530861    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6\": container with ID starting with 2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6 not found: ID does not exist" containerID="2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"
Sep 19 08:53:01.530941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.530883    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6"} err="failed to get container status \"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6\": rpc error: code = NotFound desc = could not find container \"2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6\": container with ID starting with 2de2335f8cf168efd00f9bfcba07f856f6c81ed5cf46c46ff393f598406d66d6 not found: ID does not exist"
Sep 19 08:53:01.533353 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:01.533334    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-nzq9g"]
Sep 19 08:53:02.180681 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:02.180647    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" path="/var/lib/kubelet/pods/348c2996-ba4b-41ba-ae7d-46f5a0751154/volumes"
Sep 19 08:53:02.444408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:02.444318    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 19 08:53:12.379818 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:12.379785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:53:14.182281 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.182252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:53:14.182677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.182519    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" containerID="cri-o://f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6" gracePeriod=30
Sep 19 08:53:14.182677 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.182560    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kube-rbac-proxy" containerID="cri-o://695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032" gracePeriod=30
Sep 19 08:53:14.195438 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.195412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:14.559864 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.559789    2566 generic.go:356] "Generic (PLEG): container finished" podID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerID="695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032" exitCode=2
Sep 19 08:53:14.559988 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:14.559871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerDied","Data":"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"}
Sep 19 08:53:15.364336 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:15.364301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:16.151512 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:16.151473    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 19 08:53:16.156867 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:16.156837    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 19 08:53:18.818091 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.818069    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:53:18.892451 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config\") pod \"289fbc40-0793-4dae-9978-ab2b8aae6f10\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") "
Sep 19 08:53:18.892567 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert\") pod \"289fbc40-0793-4dae-9978-ab2b8aae6f10\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") "
Sep 19 08:53:18.892567 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls\") pod \"289fbc40-0793-4dae-9978-ab2b8aae6f10\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") "
Sep 19 08:53:18.892567 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location\") pod \"289fbc40-0793-4dae-9978-ab2b8aae6f10\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") "
Sep 19 08:53:18.892567 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sqhk\" (UniqueName: \"kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk\") pod \"289fbc40-0793-4dae-9978-ab2b8aae6f10\" (UID: \"289fbc40-0793-4dae-9978-ab2b8aae6f10\") "
Sep 19 08:53:18.892849 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "289fbc40-0793-4dae-9978-ab2b8aae6f10" (UID: "289fbc40-0793-4dae-9978-ab2b8aae6f10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:18.892900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "289fbc40-0793-4dae-9978-ab2b8aae6f10" (UID: "289fbc40-0793-4dae-9978-ab2b8aae6f10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:18.892900 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.892870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "289fbc40-0793-4dae-9978-ab2b8aae6f10" (UID: "289fbc40-0793-4dae-9978-ab2b8aae6f10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:18.894401 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.894384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk" (OuterVolumeSpecName: "kube-api-access-2sqhk") pod "289fbc40-0793-4dae-9978-ab2b8aae6f10" (UID: "289fbc40-0793-4dae-9978-ab2b8aae6f10"). InnerVolumeSpecName "kube-api-access-2sqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:18.894515 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.894495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "289fbc40-0793-4dae-9978-ab2b8aae6f10" (UID: "289fbc40-0793-4dae-9978-ab2b8aae6f10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:18.993253 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.993230    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.993253 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.993249    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fbc40-0793-4dae-9978-ab2b8aae6f10-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.993373 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.993259    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fbc40-0793-4dae-9978-ab2b8aae6f10-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.993373 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.993267    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fbc40-0793-4dae-9978-ab2b8aae6f10-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.993373 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:18.993275    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqhk\" (UniqueName: \"kubernetes.io/projected/289fbc40-0793-4dae-9978-ab2b8aae6f10-kube-api-access-2sqhk\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:19.334236 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.334141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:19.399725 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.399691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:19.434979 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.434953    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:53:19.435215 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435204    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435233    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435239    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435244    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435263 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435273    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="storage-initializer"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435278    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="storage-initializer"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="storage-initializer"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="storage-initializer"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435331    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435338    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="348c2996-ba4b-41ba-ae7d-46f5a0751154" containerName="kserve-container"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435345    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kube-rbac-proxy"
Sep 19 08:53:19.435471 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.435350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerName="kserve-container"
Sep 19 08:53:19.438391 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.438376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.440830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.440805    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 19 08:53:19.440926 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.440826    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 19 08:53:19.447540 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.447516    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:53:19.496348 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.496322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.496463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.496360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.496463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.496379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.496463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.496448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq8z\" (UniqueName: \"kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.496569 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.496484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.579988 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.579952    2566 generic.go:356] "Generic (PLEG): container finished" podID="289fbc40-0793-4dae-9978-ab2b8aae6f10" containerID="f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6" exitCode=0
Sep 19 08:53:19.580130 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.580006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"
Sep 19 08:53:19.580130 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.580058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerDied","Data":"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"}
Sep 19 08:53:19.580130 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.580120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p" event={"ID":"289fbc40-0793-4dae-9978-ab2b8aae6f10","Type":"ContainerDied","Data":"3170a8e7c3e774ac6301525e9d30f0e3c9fa81c03581566bf7730e3d1275b492"}
Sep 19 08:53:19.580287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.580157    2566 scope.go:117] "RemoveContainer" containerID="695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"
Sep 19 08:53:19.588087 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.588072    2566 scope.go:117] "RemoveContainer" containerID="f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"
Sep 19 08:53:19.595451 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.595436    2566 scope.go:117] "RemoveContainer" containerID="433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"
Sep 19 08:53:19.597483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.597582 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq8z\" (UniqueName: \"kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.597582 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.597699 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.597699 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.597825 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:19.597783    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:53:19.597880 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:19.597848    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls podName:90d6d6d4-a00b-4498-a02c-436b23f5c147 nodeName:}" failed. No retries permitted until 2026-09-19 08:53:20.097832779 +0000 UTC m=+1964.418504367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls") pod "isvc-canary-promote-predictor-6546f8697f-lbs2w" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:53:19.597880 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.597868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.598297 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.598275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.598384 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.598360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.602518 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.602499    2566 scope.go:117] "RemoveContainer" containerID="695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"
Sep 19 08:53:19.602803 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:19.602785    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032\": container with ID starting with 695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032 not found: ID does not exist" containerID="695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"
Sep 19 08:53:19.602888 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.602814    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032"} err="failed to get container status \"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032\": rpc error: code = NotFound desc = could not find container \"695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032\": container with ID starting with 695734af598ae9be091c77e1755af8f28f9ca4218ad82d796f82b1db566b4032 not found: ID does not exist"
Sep 19 08:53:19.602888 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.602836    2566 scope.go:117] "RemoveContainer" containerID="f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"
Sep 19 08:53:19.603075 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:19.603058    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6\": container with ID starting with f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6 not found: ID does not exist" containerID="f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"
Sep 19 08:53:19.603114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.603082    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6"} err="failed to get container status \"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6\": rpc error: code = NotFound desc = could not find container \"f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6\": container with ID starting with f71c177af322a60da06391ce21fb179251c121feca21f1fe680de9139e11a0c6 not found: ID does not exist"
Sep 19 08:53:19.603114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.603097    2566 scope.go:117] "RemoveContainer" containerID="433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"
Sep 19 08:53:19.603193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.603129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:53:19.603332 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:19.603309    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5\": container with ID starting with 433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5 not found: ID does not exist" containerID="433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"
Sep 19 08:53:19.603369 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.603342    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5"} err="failed to get container status \"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5\": rpc error: code = NotFound desc = could not find container \"433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5\": container with ID starting with 433117b61afcf64269e65d5aef1981e40629ce4387de1c8a67e696fd113055f5 not found: ID does not exist"
Sep 19 08:53:19.605600 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.605575    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-qmv8p"]
Sep 19 08:53:19.606483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.606460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq8z\" (UniqueName: \"kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:19.613895 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.613872    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:19.765132 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.765103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:19.869658 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.869628    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:53:19.870114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.869933    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kserve-container" containerID="cri-o://e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6" gracePeriod=30
Sep 19 08:53:19.870114 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.869994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kube-rbac-proxy" containerID="cri-o://5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67" gracePeriod=30
Sep 19 08:53:19.889455 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:19.889416    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:20.102923 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.102889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:20.105199 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.105171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-lbs2w\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:20.180307 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.180245    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289fbc40-0793-4dae-9978-ab2b8aae6f10" path="/var/lib/kubelet/pods/289fbc40-0793-4dae-9978-ab2b8aae6f10/volumes"
Sep 19 08:53:20.348146 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.348115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:20.588239 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.588153    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerID="5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67" exitCode=2
Sep 19 08:53:20.588239 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.588204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerDied","Data":"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"}
Sep 19 08:53:20.619535 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:20.619506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:53:20.622496 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:53:20.622468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d6d6d4_a00b_4498_a02c_436b23f5c147.slice/crio-87285431dac6603a7456f7a15b708af7c6adb0dc181d8391cd02074f6bad6096 WatchSource:0}: Error finding container 87285431dac6603a7456f7a15b708af7c6adb0dc181d8391cd02074f6bad6096: Status 404 returned error can't find the container with id 87285431dac6603a7456f7a15b708af7c6adb0dc181d8391cd02074f6bad6096
Sep 19 08:53:21.593646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:21.593611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerStarted","Data":"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"}
Sep 19 08:53:21.594027 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:21.593667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerStarted","Data":"87285431dac6603a7456f7a15b708af7c6adb0dc181d8391cd02074f6bad6096"}
Sep 19 08:53:22.373295 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:22.373253    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:53:24.611786 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:24.611738    2566 generic.go:356] "Generic (PLEG): container finished" podID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerID="59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d" exitCode=0
Sep 19 08:53:24.612209 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:24.611800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerDied","Data":"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"}
Sep 19 08:53:24.977473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:24.977400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:25.163358 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:25.163327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:53:25.617723 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:25.617686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerStarted","Data":"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"}
Sep 19 08:53:25.618169 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:25.617749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerStarted","Data":"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"}
Sep 19 08:53:25.618169 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:25.617966    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:25.636127 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:25.636087    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podStartSLOduration=3.019106704 podStartE2EDuration="6.636074869s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.616968165s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:25.635223137 +0000 UTC m=+1969.955894724" watchObservedRunningTime="2026-09-19 08:53:25.636074869 +0000 UTC m=+1969.956746456"
Sep 19 08:53:26.606691 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.606668    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:53:26.623109 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623089    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerID="e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6" exitCode=0
Sep 19 08:53:26.623396 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623135    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"
Sep 19 08:53:26.623396 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerDied","Data":"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"}
Sep 19 08:53:26.623396 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf" event={"ID":"9d47f839-9e02-4747-a0ac-840279bf5bca","Type":"ContainerDied","Data":"60a615fe15b7e5789f363f85891651edadf53bd03ba9a103a85fece85c8cbfa9"}
Sep 19 08:53:26.623396 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623270    2566 scope.go:117] "RemoveContainer" containerID="5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"
Sep 19 08:53:26.623618 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.623600    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:26.624638 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.624614    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:53:26.632404 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.632384    2566 scope.go:117] "RemoveContainer" containerID="e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"
Sep 19 08:53:26.639193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.639172    2566 scope.go:117] "RemoveContainer" containerID="4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"
Sep 19 08:53:26.645734 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.645718    2566 scope.go:117] "RemoveContainer" containerID="5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"
Sep 19 08:53:26.646044 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:26.646024    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67\": container with ID starting with 5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67 not found: ID does not exist" containerID="5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"
Sep 19 08:53:26.646134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.646052    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67"} err="failed to get container status \"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67\": rpc error: code = NotFound desc = could not find container \"5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67\": container with ID starting with 5978b5a50d9a08d1ae0270cb0e1bff8c5e975e78546045e418138d081a47bc67 not found: ID does not exist"
Sep 19 08:53:26.646134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.646067    2566 scope.go:117] "RemoveContainer" containerID="e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"
Sep 19 08:53:26.646340 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:26.646322    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6\": container with ID starting with e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6 not found: ID does not exist" containerID="e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"
Sep 19 08:53:26.646408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.646346    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6"} err="failed to get container status \"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6\": rpc error: code = NotFound desc = could not find container \"e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6\": container with ID starting with e2360e6f4dae42334d15ce998d1024a5cdc16d91c6415a413bd38fcdec9b58c6 not found: ID does not exist"
Sep 19 08:53:26.646408 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.646367    2566 scope.go:117] "RemoveContainer" containerID="4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"
Sep 19 08:53:26.646593 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:53:26.646577    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757\": container with ID starting with 4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757 not found: ID does not exist" containerID="4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"
Sep 19 08:53:26.646655 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.646596    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757"} err="failed to get container status \"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757\": rpc error: code = NotFound desc = could not find container \"4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757\": container with ID starting with 4ab329134e7b482e5df26af40a086d9d35cec03ff359fadb8488a7913caad757 not found: ID does not exist"
Sep 19 08:53:26.751170 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xgzl\" (UniqueName: \"kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl\") pod \"9d47f839-9e02-4747-a0ac-840279bf5bca\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") "
Sep 19 08:53:26.751170 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert\") pod \"9d47f839-9e02-4747-a0ac-840279bf5bca\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") "
Sep 19 08:53:26.751312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls\") pod \"9d47f839-9e02-4747-a0ac-840279bf5bca\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") "
Sep 19 08:53:26.751312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config\") pod \"9d47f839-9e02-4747-a0ac-840279bf5bca\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") "
Sep 19 08:53:26.751312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location\") pod \"9d47f839-9e02-4747-a0ac-840279bf5bca\" (UID: \"9d47f839-9e02-4747-a0ac-840279bf5bca\") "
Sep 19 08:53:26.751820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751553    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d47f839-9e02-4747-a0ac-840279bf5bca" (UID: "9d47f839-9e02-4747-a0ac-840279bf5bca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:26.751820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d47f839-9e02-4747-a0ac-840279bf5bca" (UID: "9d47f839-9e02-4747-a0ac-840279bf5bca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:26.751820 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.751584    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d47f839-9e02-4747-a0ac-840279bf5bca" (UID: "9d47f839-9e02-4747-a0ac-840279bf5bca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:26.753272 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.753247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d47f839-9e02-4747-a0ac-840279bf5bca" (UID: "9d47f839-9e02-4747-a0ac-840279bf5bca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:26.753503 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.753480    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl" (OuterVolumeSpecName: "kube-api-access-8xgzl") pod "9d47f839-9e02-4747-a0ac-840279bf5bca" (UID: "9d47f839-9e02-4747-a0ac-840279bf5bca"). InnerVolumeSpecName "kube-api-access-8xgzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:26.852488 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.852466    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgzl\" (UniqueName: \"kubernetes.io/projected/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-api-access-8xgzl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:26.852488 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.852487    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:26.852598 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.852496    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d47f839-9e02-4747-a0ac-840279bf5bca-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:26.852598 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.852505    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d47f839-9e02-4747-a0ac-840279bf5bca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:26.852598 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.852524    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d47f839-9e02-4747-a0ac-840279bf5bca-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:53:26.944015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.943991    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:53:26.950720 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:26.950697    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2kkhf"]
Sep 19 08:53:27.627926 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:27.627890    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:53:28.179985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:28.179913    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" path="/var/lib/kubelet/pods/9d47f839-9e02-4747-a0ac-840279bf5bca/volumes"
Sep 19 08:53:32.632345 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:32.632316    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:53:32.632823 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:32.632795    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:53:42.632910 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:42.632862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:53:52.632885 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:53:52.632848    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:02.633302 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:02.633257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:12.633749 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:12.633706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:13.985847 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:13.985811    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:54:22.632726 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:22.632690    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:32.633450 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:32.633412    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:42.633913 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:42.633884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:54:49.425142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.425107    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:54:49.425542 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.425339    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" containerID="cri-o://1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4" gracePeriod=30
Sep 19 08:54:49.425594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.425525    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kube-rbac-proxy" containerID="cri-o://c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07" gracePeriod=30
Sep 19 08:54:49.453428 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.453404    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:54:49.476578 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476555    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:54:49.476828 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kube-rbac-proxy"
Sep 19 08:54:49.476877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476829    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kube-rbac-proxy"
Sep 19 08:54:49.476877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="storage-initializer"
Sep 19 08:54:49.476877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476850    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="storage-initializer"
Sep 19 08:54:49.476877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476856    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kserve-container"
Sep 19 08:54:49.476877 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476862    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kserve-container"
Sep 19 08:54:49.477030 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476907    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kserve-container"
Sep 19 08:54:49.477030 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.476912    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d47f839-9e02-4747-a0ac-840279bf5bca" containerName="kube-rbac-proxy"
Sep 19 08:54:49.479954 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.479939    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.482318 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.482302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 19 08:54:49.488953 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.488935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:54:49.565859 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.565837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.565935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.565867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4pw\" (UniqueName: \"kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.565935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.565910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.565935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.565926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.566040 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.565939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.666962 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.666940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.666966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.666985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.667026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667226 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.667186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4pw\" (UniqueName: \"kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667415 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.667394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667610 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.667589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.667666 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.667616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.669250 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.669232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.675557 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.675513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4pw\" (UniqueName: \"kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.789673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.789648    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:49.933581 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.933516    2566 generic.go:356] "Generic (PLEG): container finished" podID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerID="c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07" exitCode=2
Sep 19 08:54:49.933581 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:49.933569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerDied","Data":"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"}
Sep 19 08:54:50.064591 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:50.064562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:54:50.067423 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:54:50.067389    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc85129_ea9e_43ec_bbbb_652e5ed0fef1.slice/crio-fc93d3a17c195b6bc9a19b0c648b6d9d5620764ec165f64cf2cd3614a35b16f1 WatchSource:0}: Error finding container fc93d3a17c195b6bc9a19b0c648b6d9d5620764ec165f64cf2cd3614a35b16f1: Status 404 returned error can't find the container with id fc93d3a17c195b6bc9a19b0c648b6d9d5620764ec165f64cf2cd3614a35b16f1
Sep 19 08:54:50.939416 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:50.939381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerStarted","Data":"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"}
Sep 19 08:54:50.939819 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:50.939447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerStarted","Data":"fc93d3a17c195b6bc9a19b0c648b6d9d5620764ec165f64cf2cd3614a35b16f1"}
Sep 19 08:54:52.629043 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:52.628999    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 19 08:54:52.633344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:52.633321    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:54:53.857880 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.857858    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:54:53.952097 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.952029    2566 generic.go:356] "Generic (PLEG): container finished" podID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerID="1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4" exitCode=0
Sep 19 08:54:53.952097 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.952087    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"
Sep 19 08:54:53.952246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.952116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerDied","Data":"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"}
Sep 19 08:54:53.952246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.952217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w" event={"ID":"90d6d6d4-a00b-4498-a02c-436b23f5c147","Type":"ContainerDied","Data":"87285431dac6603a7456f7a15b708af7c6adb0dc181d8391cd02074f6bad6096"}
Sep 19 08:54:53.952322 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.952246    2566 scope.go:117] "RemoveContainer" containerID="c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"
Sep 19 08:54:53.959563 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.959545    2566 scope.go:117] "RemoveContainer" containerID="1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"
Sep 19 08:54:53.966111 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.966091    2566 scope.go:117] "RemoveContainer" containerID="59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"
Sep 19 08:54:53.972473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.972456    2566 scope.go:117] "RemoveContainer" containerID="c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"
Sep 19 08:54:53.972726 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:54:53.972709    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07\": container with ID starting with c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07 not found: ID does not exist" containerID="c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"
Sep 19 08:54:53.972822 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.972738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07"} err="failed to get container status \"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07\": rpc error: code = NotFound desc = could not find container \"c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07\": container with ID starting with c0431c89d7994642bca90711b6046951df6679d09b83e9b76edfc42d16d6fd07 not found: ID does not exist"
Sep 19 08:54:53.972822 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.972762    2566 scope.go:117] "RemoveContainer" containerID="1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"
Sep 19 08:54:53.973052 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:54:53.973035    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4\": container with ID starting with 1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4 not found: ID does not exist" containerID="1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"
Sep 19 08:54:53.973094 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.973059    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4"} err="failed to get container status \"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4\": rpc error: code = NotFound desc = could not find container \"1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4\": container with ID starting with 1b169773995ec4b45bc89e5be058cdfe8e4584cb0d476711894c37057ae016f4 not found: ID does not exist"
Sep 19 08:54:53.973094 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.973073    2566 scope.go:117] "RemoveContainer" containerID="59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"
Sep 19 08:54:53.973301 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:54:53.973288    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d\": container with ID starting with 59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d not found: ID does not exist" containerID="59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"
Sep 19 08:54:53.973343 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:53.973303    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d"} err="failed to get container status \"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d\": rpc error: code = NotFound desc = could not find container \"59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d\": container with ID starting with 59d319ce0dbe433e698773e548d34419254e7a6afd6d4ead78bc279b4a79dc8d not found: ID does not exist"
Sep 19 08:54:54.000721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location\") pod \"90d6d6d4-a00b-4498-a02c-436b23f5c147\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") "
Sep 19 08:54:54.000815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000757    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") pod \"90d6d6d4-a00b-4498-a02c-436b23f5c147\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") "
Sep 19 08:54:54.000815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000799    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert\") pod \"90d6d6d4-a00b-4498-a02c-436b23f5c147\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") "
Sep 19 08:54:54.000902 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config\") pod \"90d6d6d4-a00b-4498-a02c-436b23f5c147\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") "
Sep 19 08:54:54.000902 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qq8z\" (UniqueName: \"kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z\") pod \"90d6d6d4-a00b-4498-a02c-436b23f5c147\" (UID: \"90d6d6d4-a00b-4498-a02c-436b23f5c147\") "
Sep 19 08:54:54.001020 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.000996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90d6d6d4-a00b-4498-a02c-436b23f5c147" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:54.001190 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.001167    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90d6d6d4-a00b-4498-a02c-436b23f5c147-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:54:54.001249 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.001198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90d6d6d4-a00b-4498-a02c-436b23f5c147" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:54:54.001296 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.001242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90d6d6d4-a00b-4498-a02c-436b23f5c147" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:54:54.002658 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.002637    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90d6d6d4-a00b-4498-a02c-436b23f5c147" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:54:54.002888 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.002868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z" (OuterVolumeSpecName: "kube-api-access-5qq8z") pod "90d6d6d4-a00b-4498-a02c-436b23f5c147" (UID: "90d6d6d4-a00b-4498-a02c-436b23f5c147"). InnerVolumeSpecName "kube-api-access-5qq8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:54:54.102107 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.102081    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:54:54.102107 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.102104    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq8z\" (UniqueName: \"kubernetes.io/projected/90d6d6d4-a00b-4498-a02c-436b23f5c147-kube-api-access-5qq8z\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:54:54.102242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.102114    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90d6d6d4-a00b-4498-a02c-436b23f5c147-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:54:54.102242 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.102122    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90d6d6d4-a00b-4498-a02c-436b23f5c147-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:54:54.267829 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.267761    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:54:54.273488 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.273469    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-lbs2w"]
Sep 19 08:54:54.958306 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.958228    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerID="ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9" exitCode=0
Sep 19 08:54:54.958306 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:54.958272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerDied","Data":"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"}
Sep 19 08:54:55.750245 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.750211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:54:55.963665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.963635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerStarted","Data":"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"}
Sep 19 08:54:55.964036 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.963691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerStarted","Data":"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"}
Sep 19 08:54:55.964036 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.963949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:55.964036 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.964007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:54:55.965066 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.965041    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:54:55.982329 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:55.982289    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podStartSLOduration=3.054099712 podStartE2EDuration="6.982274526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.928174814s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:54:55.981386777 +0000 UTC m=+2060.302058363" watchObservedRunningTime="2026-09-19 08:54:55.982274526 +0000 UTC m=+2060.302946114"
Sep 19 08:54:56.180306 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:56.180284    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" path="/var/lib/kubelet/pods/90d6d6d4-a00b-4498-a02c-436b23f5c147/volumes"
Sep 19 08:54:56.967317 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:54:56.967273    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:00.883015 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.882939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:55:00.916763 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.916731    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:55:00.917094 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kube-rbac-proxy"
Sep 19 08:55:00.917178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917097    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kube-rbac-proxy"
Sep 19 08:55:00.917178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917110    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container"
Sep 19 08:55:00.917178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917118    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container"
Sep 19 08:55:00.917178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917138    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="storage-initializer"
Sep 19 08:55:00.917178 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917148    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="storage-initializer"
Sep 19 08:55:00.917431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kube-rbac-proxy"
Sep 19 08:55:00.917431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.917226    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d6d6d4-a00b-4498-a02c-436b23f5c147" containerName="kserve-container"
Sep 19 08:55:00.921703 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.921682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:00.924001 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.923978    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 19 08:55:00.924093 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.923983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 19 08:55:00.929295 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:00.929269    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:55:01.049827 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.049791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.049950 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.049849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.049950 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.049939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.050037 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.049966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.050037 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.049985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsqz\" (UniqueName: \"kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150490 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150490 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150490 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsqz\" (UniqueName: \"kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.150706 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:55:01.150625    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 19 08:55:01.150706 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:55:01.150695    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls podName:3d656e7c-8e69-46ec-b1d1-6b4fecf71618 nodeName:}" failed. No retries permitted until 2026-09-19 08:55:01.6506742 +0000 UTC m=+2065.971345770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 19 08:55:01.150964 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.150944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.151191 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.151176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.151232 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.151200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.159123 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.159106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsqz\" (UniqueName: \"kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.441985 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.441904    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:55:01.653235 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.653203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.655482 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.655460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.832120 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.832055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:01.971287 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.971262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:55:01.971903 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:01.971876    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:02.103536 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:02.103517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:55:02.105290 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:55:02.105265    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d656e7c_8e69_46ec_b1d1_6b4fecf71618.slice/crio-3bc9e28819a1e04fee68fb2e1916080fa580fd5b4ea996c15a33d2a48143877c WatchSource:0}: Error finding container 3bc9e28819a1e04fee68fb2e1916080fa580fd5b4ea996c15a33d2a48143877c: Status 404 returned error can't find the container with id 3bc9e28819a1e04fee68fb2e1916080fa580fd5b4ea996c15a33d2a48143877c
Sep 19 08:55:02.990702 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:02.990666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerStarted","Data":"8d50344582df570a9d3833fa17b0af262d0addfce169e0099fa36705675fb63b"}
Sep 19 08:55:02.991099 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:02.990725    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerStarted","Data":"3bc9e28819a1e04fee68fb2e1916080fa580fd5b4ea996c15a33d2a48143877c"}
Sep 19 08:55:06.004587 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:06.004557    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerID="8d50344582df570a9d3833fa17b0af262d0addfce169e0099fa36705675fb63b" exitCode=0
Sep 19 08:55:06.004966 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:06.004633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerDied","Data":"8d50344582df570a9d3833fa17b0af262d0addfce169e0099fa36705675fb63b"}
Sep 19 08:55:07.010115 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:07.010081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerStarted","Data":"bf1f7b014bd73c334d93eaf61fdea968d885fd9f9ee8d5ef82a5ee43988b4e56"}
Sep 19 08:55:07.010572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:07.010150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerStarted","Data":"f3cf1b5d1519b21044cb72ce3cf0f36ca50544cfe36b300100234a7d525437c1"}
Sep 19 08:55:07.010572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:07.010364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:07.031386 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:07.031344    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podStartSLOduration=3.305411433 podStartE2EDuration="7.031332246s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725920813s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:55:07.030344528 +0000 UTC m=+2071.351016115" watchObservedRunningTime="2026-09-19 08:55:07.031332246 +0000 UTC m=+2071.352003833"
Sep 19 08:55:08.014009 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:08.013977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:08.015160 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:08.015133    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:09.017392 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:09.017344    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:11.971849 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:11.971814    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:14.022090 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:14.022052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:55:14.022706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:14.022674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:21.972459 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:21.972416    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:24.023255 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:24.023216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:31.971889 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:31.971850    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:34.023658 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:34.023615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:41.971755 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:41.971717    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:42.607796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:42.607748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:55:44.022607 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:44.022569    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:55:51.972478 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:51.972441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:55:54.023501 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:55:54.023459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:56:01.972120 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:01.972080    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:56:04.022826 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:04.022762    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:56:11.972442 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:11.972417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:56:14.022948 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:14.022909    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:56:19.176658 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:19.176626    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:56:19.587122 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:19.587047    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:56:19.587377 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:19.587353    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" containerID="cri-o://87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d" gracePeriod=30
Sep 19 08:56:19.587434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:19.587408    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kube-rbac-proxy" containerID="cri-o://849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04" gracePeriod=30
Sep 19 08:56:19.611848 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:19.611824    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:20.281739 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:20.281706    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerID="849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04" exitCode=2
Sep 19 08:56:20.282163 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:20.281795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerDied","Data":"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"}
Sep 19 08:56:21.669260 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.669230    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:56:21.669665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.669438    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container" containerID="cri-o://f3cf1b5d1519b21044cb72ce3cf0f36ca50544cfe36b300100234a7d525437c1" gracePeriod=30
Sep 19 08:56:21.669665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.669477    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kube-rbac-proxy" containerID="cri-o://bf1f7b014bd73c334d93eaf61fdea968d885fd9f9ee8d5ef82a5ee43988b4e56" gracePeriod=30
Sep 19 08:56:21.682557 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.682531    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:21.967961 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.967890    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 19 08:56:21.972205 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:21.972183    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:56:22.290918 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:22.290840    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerID="bf1f7b014bd73c334d93eaf61fdea968d885fd9f9ee8d5ef82a5ee43988b4e56" exitCode=2
Sep 19 08:56:22.290918 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:22.290868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerDied","Data":"bf1f7b014bd73c334d93eaf61fdea968d885fd9f9ee8d5ef82a5ee43988b4e56"}
Sep 19 08:56:24.017308 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.017283    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:56:24.017648 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.017509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 19 08:56:24.170986 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.170955    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location\") pod \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") "
Sep 19 08:56:24.171142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.170996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert\") pod \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") "
Sep 19 08:56:24.171142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz4pw\" (UniqueName: \"kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw\") pod \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") "
Sep 19 08:56:24.171142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171046    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls\") pod \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") "
Sep 19 08:56:24.171142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171085    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config\") pod \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\" (UID: \"abc85129-ea9e-43ec-bbbb-652e5ed0fef1\") "
Sep 19 08:56:24.171337 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abc85129-ea9e-43ec-bbbb-652e5ed0fef1" (UID: "abc85129-ea9e-43ec-bbbb-652e5ed0fef1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:24.171337 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abc85129-ea9e-43ec-bbbb-652e5ed0fef1" (UID: "abc85129-ea9e-43ec-bbbb-652e5ed0fef1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:24.171526 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.171505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abc85129-ea9e-43ec-bbbb-652e5ed0fef1" (UID: "abc85129-ea9e-43ec-bbbb-652e5ed0fef1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:24.173135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.173110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw" (OuterVolumeSpecName: "kube-api-access-cz4pw") pod "abc85129-ea9e-43ec-bbbb-652e5ed0fef1" (UID: "abc85129-ea9e-43ec-bbbb-652e5ed0fef1"). InnerVolumeSpecName "kube-api-access-cz4pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:24.173236 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.173137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abc85129-ea9e-43ec-bbbb-652e5ed0fef1" (UID: "abc85129-ea9e-43ec-bbbb-652e5ed0fef1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:56:24.272339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.272314    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.272339 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.272338    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.272481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.272347    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz4pw\" (UniqueName: \"kubernetes.io/projected/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-api-access-cz4pw\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.272481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.272356    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.272481 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.272364    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc85129-ea9e-43ec-bbbb-652e5ed0fef1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.300070 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.300045    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerID="87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d" exitCode=0
Sep 19 08:56:24.300155 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.300108    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"
Sep 19 08:56:24.300155 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.300124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerDied","Data":"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"}
Sep 19 08:56:24.300229 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.300181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh" event={"ID":"abc85129-ea9e-43ec-bbbb-652e5ed0fef1","Type":"ContainerDied","Data":"fc93d3a17c195b6bc9a19b0c648b6d9d5620764ec165f64cf2cd3614a35b16f1"}
Sep 19 08:56:24.300229 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.300206    2566 scope.go:117] "RemoveContainer" containerID="849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"
Sep 19 08:56:24.307690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.307674    2566 scope.go:117] "RemoveContainer" containerID="87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"
Sep 19 08:56:24.314157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.314143    2566 scope.go:117] "RemoveContainer" containerID="ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"
Sep 19 08:56:24.320639 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.320620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:56:24.321007 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.320993    2566 scope.go:117] "RemoveContainer" containerID="849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"
Sep 19 08:56:24.321245 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:24.321226    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04\": container with ID starting with 849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04 not found: ID does not exist" containerID="849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"
Sep 19 08:56:24.321313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.321256    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04"} err="failed to get container status \"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04\": rpc error: code = NotFound desc = could not find container \"849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04\": container with ID starting with 849fd53e5a59761d6141b2ec233819d729a6b660bc2ff186f8e3cc5f3a48bf04 not found: ID does not exist"
Sep 19 08:56:24.321313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.321279    2566 scope.go:117] "RemoveContainer" containerID="87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"
Sep 19 08:56:24.321539 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:24.321521    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d\": container with ID starting with 87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d not found: ID does not exist" containerID="87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"
Sep 19 08:56:24.321582 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.321545    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d"} err="failed to get container status \"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d\": rpc error: code = NotFound desc = could not find container \"87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d\": container with ID starting with 87b35241528e551ffed01b65bbff330738f5fc2930a300bc5830d5c3183f867d not found: ID does not exist"
Sep 19 08:56:24.321582 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.321559    2566 scope.go:117] "RemoveContainer" containerID="ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"
Sep 19 08:56:24.321741 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:24.321725    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9\": container with ID starting with ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9 not found: ID does not exist" containerID="ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"
Sep 19 08:56:24.321885 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.321745    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9"} err="failed to get container status \"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9\": rpc error: code = NotFound desc = could not find container \"ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9\": container with ID starting with ef5ac7c7856525db43b1dffc52085b81b3872cfd98b9b569a5624b80264fe2e9 not found: ID does not exist"
Sep 19 08:56:24.331095 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.331074    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-sl8zh"]
Sep 19 08:56:24.751581 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.751545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:24.827956 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.827923    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:24.870022 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.869991    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:56:24.870259 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870247    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container"
Sep 19 08:56:24.870313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870261    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container"
Sep 19 08:56:24.870313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870272    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.870313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870278    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.870313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870285    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="storage-initializer"
Sep 19 08:56:24.870313 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="storage-initializer"
Sep 19 08:56:24.870483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870363    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kserve-container"
Sep 19 08:56:24.870483 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.870373    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.874690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.874674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:24.876899 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.876875    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 19 08:56:24.877014 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.876908    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 19 08:56:24.884020 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.884001    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:56:24.977588 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.977560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:24.977753 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.977602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:24.977753 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.977630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:24.977753 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.977694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:24.977950 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:24.977761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vxq\" (UniqueName: \"kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.015406 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.015332    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:25.079154 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vxq\" (UniqueName: \"kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079473 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079473 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:25.079413    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:56:25.079657 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:25.079480    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls podName:e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f nodeName:}" failed. No retries permitted until 2026-09-19 08:56:25.579458797 +0000 UTC m=+2149.900130379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-fv7p6" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:56:25.079706 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079854 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.079937 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.079919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.087690 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.087671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vxq\" (UniqueName: \"kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.137633 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.137612    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:25.583809 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.583760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.586201 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.586170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-fv7p6\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:25.784656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:25.784626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:26.063986 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.063955    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:56:26.066429 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:56:26.066404    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dd54f2_2f1b_4de4_9be2_fca6f237ab2f.slice/crio-0a813cd39afb86098b850d80f34751d96f50ccd2087fa85ad0d3b8c354a1748e WatchSource:0}: Error finding container 0a813cd39afb86098b850d80f34751d96f50ccd2087fa85ad0d3b8c354a1748e: Status 404 returned error can't find the container with id 0a813cd39afb86098b850d80f34751d96f50ccd2087fa85ad0d3b8c354a1748e
Sep 19 08:56:26.182704 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.182668    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc85129-ea9e-43ec-bbbb-652e5ed0fef1" path="/var/lib/kubelet/pods/abc85129-ea9e-43ec-bbbb-652e5ed0fef1/volumes"
Sep 19 08:56:26.311246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.311198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerStarted","Data":"8eb01833134b75df05f38aa01e380563759b2a61e99995c5c294dd491938287b"}
Sep 19 08:56:26.311477 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.311273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerStarted","Data":"0a813cd39afb86098b850d80f34751d96f50ccd2087fa85ad0d3b8c354a1748e"}
Sep 19 08:56:26.314743 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.314678    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerID="f3cf1b5d1519b21044cb72ce3cf0f36ca50544cfe36b300100234a7d525437c1" exitCode=0
Sep 19 08:56:26.314743 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.314737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerDied","Data":"f3cf1b5d1519b21044cb72ce3cf0f36ca50544cfe36b300100234a7d525437c1"}
Sep 19 08:56:26.412410 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.412387    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:56:26.490148 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490122    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvsqz\" (UniqueName: \"kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz\") pod \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") "
Sep 19 08:56:26.490292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490182    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert\") pod \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") "
Sep 19 08:56:26.490292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490207    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") pod \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") "
Sep 19 08:56:26.490292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location\") pod \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") "
Sep 19 08:56:26.490292 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config\") pod \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\" (UID: \"3d656e7c-8e69-46ec-b1d1-6b4fecf71618\") "
Sep 19 08:56:26.490585 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490556    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d656e7c-8e69-46ec-b1d1-6b4fecf71618" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:26.490704 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d656e7c-8e69-46ec-b1d1-6b4fecf71618" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:26.490762 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.490694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d656e7c-8e69-46ec-b1d1-6b4fecf71618" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:26.492117 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.492096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz" (OuterVolumeSpecName: "kube-api-access-gvsqz") pod "3d656e7c-8e69-46ec-b1d1-6b4fecf71618" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618"). InnerVolumeSpecName "kube-api-access-gvsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:26.492606 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.492588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d656e7c-8e69-46ec-b1d1-6b4fecf71618" (UID: "3d656e7c-8e69-46ec-b1d1-6b4fecf71618"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:56:26.591465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.591411    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvsqz\" (UniqueName: \"kubernetes.io/projected/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-api-access-gvsqz\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:26.591465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.591431    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:26.591465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.591441    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:26.591465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.591450    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:26.591465 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.591458    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d656e7c-8e69-46ec-b1d1-6b4fecf71618-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:56:26.832999 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.832964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:26.892427 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892403    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:56:26.892665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892654    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="storage-initializer"
Sep 19 08:56:26.892712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892666    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="storage-initializer"
Sep 19 08:56:26.892712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892674    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kube-rbac-proxy"
Sep 19 08:56:26.892712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892679    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kube-rbac-proxy"
Sep 19 08:56:26.892712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892693    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container"
Sep 19 08:56:26.892712 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892699    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container"
Sep 19 08:56:26.892882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892737    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kserve-container"
Sep 19 08:56:26.892882 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.892745    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" containerName="kube-rbac-proxy"
Sep 19 08:56:26.895939 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.895919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:26.898364 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.898347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 19 08:56:26.898463 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.898383    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 19 08:56:26.906031 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.906012    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:56:26.995148 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.995124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:26.995284 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.995156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:26.995284 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.995182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9l8\" (UniqueName: \"kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:26.995402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.995284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:26.995402 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:26.995342    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096095 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096220 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096220 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9l8\" (UniqueName: \"kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096220 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096369 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:27.096242    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 19 08:56:27.096369 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096369 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:56:27.096303    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls podName:52110382-6ac9-4308-9d1b-efabcd4092dd nodeName:}" failed. No retries permitted until 2026-09-19 08:56:27.596283842 +0000 UTC m=+2151.916955409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-w6f62" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 19 08:56:27.096554 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096852 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.096896 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.096845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.106003 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.105982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9l8\" (UniqueName: \"kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.310873 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.310792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:56:27.320796 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.320757    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"
Sep 19 08:56:27.320941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.320807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl" event={"ID":"3d656e7c-8e69-46ec-b1d1-6b4fecf71618","Type":"ContainerDied","Data":"3bc9e28819a1e04fee68fb2e1916080fa580fd5b4ea996c15a33d2a48143877c"}
Sep 19 08:56:27.320941 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.320884    2566 scope.go:117] "RemoveContainer" containerID="bf1f7b014bd73c334d93eaf61fdea968d885fd9f9ee8d5ef82a5ee43988b4e56"
Sep 19 08:56:27.331643 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.331622    2566 scope.go:117] "RemoveContainer" containerID="f3cf1b5d1519b21044cb72ce3cf0f36ca50544cfe36b300100234a7d525437c1"
Sep 19 08:56:27.339102 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.339081    2566 scope.go:117] "RemoveContainer" containerID="8d50344582df570a9d3833fa17b0af262d0addfce169e0099fa36705675fb63b"
Sep 19 08:56:27.349839 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.349810    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:56:27.355974 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.355951    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-6jkcl"]
Sep 19 08:56:27.600608 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.600535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.602879 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.602858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w6f62\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:27.804854 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:27.804823    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:28.075231 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:28.075208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:56:28.077016 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:56:28.076992    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52110382_6ac9_4308_9d1b_efabcd4092dd.slice/crio-8e8ce2ea5b05bc87c36a9b872da2116e029fce240e61d8e8b922220ea5c985e4 WatchSource:0}: Error finding container 8e8ce2ea5b05bc87c36a9b872da2116e029fce240e61d8e8b922220ea5c985e4: Status 404 returned error can't find the container with id 8e8ce2ea5b05bc87c36a9b872da2116e029fce240e61d8e8b922220ea5c985e4
Sep 19 08:56:28.180523 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:28.180447    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d656e7c-8e69-46ec-b1d1-6b4fecf71618" path="/var/lib/kubelet/pods/3d656e7c-8e69-46ec-b1d1-6b4fecf71618/volumes"
Sep 19 08:56:28.327060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:28.327023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerStarted","Data":"38d6ebfce8125c5ad7eea8a19adf060fa3a546fc4df587d66c5ee327a4892d53"}
Sep 19 08:56:28.327467 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:28.327091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerStarted","Data":"8e8ce2ea5b05bc87c36a9b872da2116e029fce240e61d8e8b922220ea5c985e4"}
Sep 19 08:56:30.334800 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:30.334754    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerID="8eb01833134b75df05f38aa01e380563759b2a61e99995c5c294dd491938287b" exitCode=0
Sep 19 08:56:30.335161 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:30.334830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerDied","Data":"8eb01833134b75df05f38aa01e380563759b2a61e99995c5c294dd491938287b"}
Sep 19 08:56:31.340416 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:31.340387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerStarted","Data":"a3435a83439d058f5a16c5dbadd5c7814c6d61a3b54d2e28112b36bbb3340593"}
Sep 19 08:56:31.340761 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:31.340444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerStarted","Data":"6b01ff38369e150ede0419266b8168541a90a27eb4c13b02213f5f73d8b98f2d"}
Sep 19 08:56:31.340761 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:31.340622    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:31.362936 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:31.362890    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podStartSLOduration=3.627344762 podStartE2EDuration="7.362855154s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.735510392s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:31.359117825 +0000 UTC m=+2155.679789412" watchObservedRunningTime="2026-09-19 08:56:31.362855154 +0000 UTC m=+2155.683526796"
Sep 19 08:56:32.344739 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:32.344709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:32.346392 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:32.346363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:56:33.350165 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:33.350123    2566 generic.go:356] "Generic (PLEG): container finished" podID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerID="38d6ebfce8125c5ad7eea8a19adf060fa3a546fc4df587d66c5ee327a4892d53" exitCode=0
Sep 19 08:56:33.350594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:33.350186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerDied","Data":"38d6ebfce8125c5ad7eea8a19adf060fa3a546fc4df587d66c5ee327a4892d53"}
Sep 19 08:56:33.350594 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:33.350546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:56:33.351644 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:33.351623    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:56:37.370239 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:37.370205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerStarted","Data":"f83c954bbd89bf7fd1145aae87d1289f683676052de8c3114c4557fa51088df9"}
Sep 19 08:56:37.370590 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:37.370265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerStarted","Data":"8ef7fb7b1dd0f5de8abc287ac7f306572bf50143984605a20bc17fb4f195755a"}
Sep 19 08:56:37.370590 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:37.370494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:37.392029 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:37.391988    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podStartSLOduration=3.532851215 podStartE2EDuration="11.391975761s" totalImagesPullingTime="3.519595597s" totalInitContainerRuntime="4.339528949s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:37.390403031 +0000 UTC m=+2161.711074618" watchObservedRunningTime="2026-09-19 08:56:37.391975761 +0000 UTC m=+2161.712647348"
Sep 19 08:56:38.354654 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:38.354627    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:56:38.355196 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:38.355165    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:56:38.373953 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:38.373926    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:38.374919 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:38.374899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:56:39.382165 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:39.382122    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:56:44.385034 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:44.385006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:44.385674 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:44.385644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:56:48.355167 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:48.355130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:56:54.385870 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:54.385840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:56:58.355842 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:56:58.355799    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:06.203800 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:06.203746    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:06.304579 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:06.304548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:07.158246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.158209    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:57:07.158516 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.158491    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container" containerID="cri-o://8ef7fb7b1dd0f5de8abc287ac7f306572bf50143984605a20bc17fb4f195755a" gracePeriod=30
Sep 19 08:57:07.158596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.158516    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" containerID="cri-o://f83c954bbd89bf7fd1145aae87d1289f683676052de8c3114c4557fa51088df9" gracePeriod=30
Sep 19 08:57:07.182846 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.182822    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:07.489741 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.489656    2566 generic.go:356] "Generic (PLEG): container finished" podID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerID="f83c954bbd89bf7fd1145aae87d1289f683676052de8c3114c4557fa51088df9" exitCode=2
Sep 19 08:57:07.489741 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:07.489720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerDied","Data":"f83c954bbd89bf7fd1145aae87d1289f683676052de8c3114c4557fa51088df9"}
Sep 19 08:57:08.355421 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:08.355387    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:09.381889 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:09.381841    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:12.319857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:12.319824    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:12.767336 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:12.767304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:14.381916 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:14.381878    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:18.355698 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:18.355652    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:19.381752 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:19.381713    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:19.382145 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:19.381868    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:57:24.381389 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:24.381347    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:28.355037 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:28.354996    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:29.381308 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:29.381266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:34.381385 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:34.381348    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:57:37.604878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.604789    2566 generic.go:356] "Generic (PLEG): container finished" podID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerID="8ef7fb7b1dd0f5de8abc287ac7f306572bf50143984605a20bc17fb4f195755a" exitCode=137
Sep 19 08:57:37.604878 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.604805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerDied","Data":"8ef7fb7b1dd0f5de8abc287ac7f306572bf50143984605a20bc17fb4f195755a"}
Sep 19 08:57:37.799186 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.799163    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:57:37.953718 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.953693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config\") pod \"52110382-6ac9-4308-9d1b-efabcd4092dd\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") "
Sep 19 08:57:37.953894 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.953743    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") pod \"52110382-6ac9-4308-9d1b-efabcd4092dd\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") "
Sep 19 08:57:37.953894 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.953779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert\") pod \"52110382-6ac9-4308-9d1b-efabcd4092dd\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") "
Sep 19 08:57:37.953894 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.953800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location\") pod \"52110382-6ac9-4308-9d1b-efabcd4092dd\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") "
Sep 19 08:57:37.954060 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.953934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn9l8\" (UniqueName: \"kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8\") pod \"52110382-6ac9-4308-9d1b-efabcd4092dd\" (UID: \"52110382-6ac9-4308-9d1b-efabcd4092dd\") "
Sep 19 08:57:37.954134 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.954109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52110382-6ac9-4308-9d1b-efabcd4092dd" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:37.954199 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.954159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52110382-6ac9-4308-9d1b-efabcd4092dd" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:37.954257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.954222    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:57:37.954257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.954234    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52110382-6ac9-4308-9d1b-efabcd4092dd-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:57:37.956001 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.955973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8" (OuterVolumeSpecName: "kube-api-access-dn9l8") pod "52110382-6ac9-4308-9d1b-efabcd4092dd" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd"). InnerVolumeSpecName "kube-api-access-dn9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:37.956102 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.956010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52110382-6ac9-4308-9d1b-efabcd4092dd" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:37.964589 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:37.964565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52110382-6ac9-4308-9d1b-efabcd4092dd" (UID: "52110382-6ac9-4308-9d1b-efabcd4092dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:38.055175 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.055153    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52110382-6ac9-4308-9d1b-efabcd4092dd-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:57:38.055175 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.055174    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52110382-6ac9-4308-9d1b-efabcd4092dd-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:57:38.055303 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.055185    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9l8\" (UniqueName: \"kubernetes.io/projected/52110382-6ac9-4308-9d1b-efabcd4092dd-kube-api-access-dn9l8\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:57:38.355266 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.355189    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:38.610519 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.610494    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"
Sep 19 08:57:38.610519 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.610510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62" event={"ID":"52110382-6ac9-4308-9d1b-efabcd4092dd","Type":"ContainerDied","Data":"8e8ce2ea5b05bc87c36a9b872da2116e029fce240e61d8e8b922220ea5c985e4"}
Sep 19 08:57:38.611008 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.610570    2566 scope.go:117] "RemoveContainer" containerID="f83c954bbd89bf7fd1145aae87d1289f683676052de8c3114c4557fa51088df9"
Sep 19 08:57:38.618257 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.618237    2566 scope.go:117] "RemoveContainer" containerID="8ef7fb7b1dd0f5de8abc287ac7f306572bf50143984605a20bc17fb4f195755a"
Sep 19 08:57:38.625193 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.625177    2566 scope.go:117] "RemoveContainer" containerID="38d6ebfce8125c5ad7eea8a19adf060fa3a546fc4df587d66c5ee327a4892d53"
Sep 19 08:57:38.643680 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.643660    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:57:38.656871 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:38.656852    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w6f62"]
Sep 19 08:57:40.180157 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:40.180121    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" path="/var/lib/kubelet/pods/52110382-6ac9-4308-9d1b-efabcd4092dd/volumes"
Sep 19 08:57:41.175818 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:41.175759    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:57:51.176520 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:51.176488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:57:51.377657 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:51.377618    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:54.824609 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:54.824574    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:56.453921 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.453886    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:56.482135 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 08:57:56.482367 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482355    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy"
Sep 19 08:57:56.482413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482368    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy"
Sep 19 08:57:56.482413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482383    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container"
Sep 19 08:57:56.482413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482389    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container"
Sep 19 08:57:56.482413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482401    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="storage-initializer"
Sep 19 08:57:56.482413 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482407    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="storage-initializer"
Sep 19 08:57:56.482605 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482445    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kube-rbac-proxy"
Sep 19 08:57:56.482605 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.482455    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52110382-6ac9-4308-9d1b-efabcd4092dd" containerName="kserve-container"
Sep 19 08:57:56.485435 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.485419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.487601 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.487576    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 19 08:57:56.487721 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.487695    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 19 08:57:56.494577 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.494558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 08:57:56.590532 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.590507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.590683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.590552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.590683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.590573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnk4t\" (UniqueName: \"kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.590683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.590646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.590683 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.590664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.691732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.691704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.691732 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.691736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.691949 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.691759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.691949 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.691808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.691949 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.691824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnk4t\" (UniqueName: \"kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.692076 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:57:56.691988    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 19 08:57:56.692076 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:57:56.692066    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls podName:738fb929-79c1-4868-8b52-380fd2d62a8d nodeName:}" failed. No retries permitted until 2026-09-19 08:57:57.192044705 +0000 UTC m=+2241.512716292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls") pod "isvc-triton-predictor-5d7db58676-hg4zt" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d") : secret "isvc-triton-predictor-serving-cert" not found
Sep 19 08:57:56.692204 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.692114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.692452 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.692434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.692494 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.692453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.702516 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.702494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnk4t\" (UniqueName: \"kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:56.785795 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:56.785698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:57:57.195459 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:57.195430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:57.197809 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:57.197763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-hg4zt\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:57.395085 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:57.395056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:57:57.663857 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:57.663832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 08:57:57.666333 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:57:57.666306    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738fb929_79c1_4868_8b52_380fd2d62a8d.slice/crio-52b66d06cd7522d24bf00926d799689a82d29ac55d98f333537ac77a3ebce44d WatchSource:0}: Error finding container 52b66d06cd7522d24bf00926d799689a82d29ac55d98f333537ac77a3ebce44d: Status 404 returned error can't find the container with id 52b66d06cd7522d24bf00926d799689a82d29ac55d98f333537ac77a3ebce44d
Sep 19 08:57:57.683596 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:57.683570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerStarted","Data":"52b66d06cd7522d24bf00926d799689a82d29ac55d98f333537ac77a3ebce44d"}
Sep 19 08:57:58.689225 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:57:58.689149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerStarted","Data":"d9c9156d6ed83ec3bc7083e299f7a5a5a17f48420e08a611df89d1eec83dcb8d"}
Sep 19 08:58:01.701629 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:58:01.701604    2566 generic.go:356] "Generic (PLEG): container finished" podID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerID="d9c9156d6ed83ec3bc7083e299f7a5a5a17f48420e08a611df89d1eec83dcb8d" exitCode=0
Sep 19 08:58:01.702010 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:58:01.701665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerDied","Data":"d9c9156d6ed83ec3bc7083e299f7a5a5a17f48420e08a611df89d1eec83dcb8d"}
Sep 19 08:59:24.922787 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:24.918752    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:59:24.922787 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:24.919691    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" containerID="cri-o://6b01ff38369e150ede0419266b8168541a90a27eb4c13b02213f5f73d8b98f2d" gracePeriod=30
Sep 19 08:59:24.922787 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:24.920059    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" containerID="cri-o://a3435a83439d058f5a16c5dbadd5c7814c6d61a3b54d2e28112b36bbb3340593" gracePeriod=30
Sep 19 08:59:24.944698 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:24.944663    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:59:28.351790 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:28.351692    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:59:30.084492 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.084394    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:59:30.175572 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.175523    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:59:30.205221 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.205138    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 08:59:30.219986 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.219954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 08:59:30.220153 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.220067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.222422 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.222397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 19 08:59:30.222560 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.222448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 19 08:59:30.335962 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.335928    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:59:30.386384 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.386354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.386543 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.386400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.386543 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.386424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.386632 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.386536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57nd\" (UniqueName: \"kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.386632 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.386604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487075 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.486994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57nd\" (UniqueName: \"kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487241 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.487075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487241 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.487108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487241 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.487137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487241 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.487158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.487447 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:59:30.487241    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 19 08:59:30.487447 ip-10-0-136-228 kubenswrapper[2566]: E0919 08:59:30.487310    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls podName:21a68574-29c5-40bc-af35-40ef669a77b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:59:30.987292643 +0000 UTC m=+2335.307964214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls") pod "isvc-canary-stop-predictor-5ffdccb94c-wx4j9" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 19 08:59:30.487681 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.487650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.488033 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.488011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.488112 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.488030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.495875 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.495841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57nd\" (UniqueName: \"kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.644427 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.644388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 08:59:30.991656 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.991616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:30.994331 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:30.994305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wx4j9\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:31.131431 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:31.131395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:31.176434 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:31.176391    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:59:33.351665 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:33.351617    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:59:38.351696 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:38.351651    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:59:38.352142 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:38.351812    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:59:41.176053 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:41.176016    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:59:43.351341 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:43.351298    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:59:48.350951 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:48.350901    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:59:51.176311 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:51.176274    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:59:51.176701 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:51.176400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:59:52.222344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.222306    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerID="a3435a83439d058f5a16c5dbadd5c7814c6d61a3b54d2e28112b36bbb3340593" exitCode=2
Sep 19 08:59:52.222344 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.222339    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerID="6b01ff38369e150ede0419266b8168541a90a27eb4c13b02213f5f73d8b98f2d" exitCode=0
Sep 19 08:59:52.222702 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.222382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerDied","Data":"a3435a83439d058f5a16c5dbadd5c7814c6d61a3b54d2e28112b36bbb3340593"}
Sep 19 08:59:52.222702 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.222434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerDied","Data":"6b01ff38369e150ede0419266b8168541a90a27eb4c13b02213f5f73d8b98f2d"}
Sep 19 08:59:52.708519 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.708410    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:59:52.779676 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.779648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location\") pod \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") "
Sep 19 08:59:52.779830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.779689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") pod \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") "
Sep 19 08:59:52.779830 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.779724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert\") pod \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") "
Sep 19 08:59:52.779935 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.779916    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4vxq\" (UniqueName: \"kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq\") pod \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") "
Sep 19 08:59:52.779994 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.779979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config\") pod \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\" (UID: \"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f\") "
Sep 19 08:59:52.780034 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.780011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:52.780073 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.780045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:59:52.780251 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.780234    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:59:52.780317 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.780259    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:59:52.780317 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.780296    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:59:52.781681 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.781664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:59:52.782005 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.781981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq" (OuterVolumeSpecName: "kube-api-access-v4vxq") pod "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" (UID: "e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f"). InnerVolumeSpecName "kube-api-access-v4vxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:59:52.881578 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.881548    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vxq\" (UniqueName: \"kubernetes.io/projected/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-api-access-v4vxq\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:59:52.881578 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.881574    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:59:52.881745 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.881586    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 08:59:52.936269 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:52.936248    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 08:59:53.055220 ip-10-0-136-228 kubenswrapper[2566]: W0919 08:59:53.055186    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a68574_29c5_40bc_af35_40ef669a77b6.slice/crio-68113984bc139dfc7aa605690a0fafda505cedb03a03fc4f84560c7c60cb9c70 WatchSource:0}: Error finding container 68113984bc139dfc7aa605690a0fafda505cedb03a03fc4f84560c7c60cb9c70: Status 404 returned error can't find the container with id 68113984bc139dfc7aa605690a0fafda505cedb03a03fc4f84560c7c60cb9c70
Sep 19 08:59:53.228815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.228735    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"
Sep 19 08:59:53.228815 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.228734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6" event={"ID":"e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f","Type":"ContainerDied","Data":"0a813cd39afb86098b850d80f34751d96f50ccd2087fa85ad0d3b8c354a1748e"}
Sep 19 08:59:53.229256 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.228863    2566 scope.go:117] "RemoveContainer" containerID="a3435a83439d058f5a16c5dbadd5c7814c6d61a3b54d2e28112b36bbb3340593"
Sep 19 08:59:53.230543 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.230512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerStarted","Data":"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"}
Sep 19 08:59:53.230646 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.230566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerStarted","Data":"68113984bc139dfc7aa605690a0fafda505cedb03a03fc4f84560c7c60cb9c70"}
Sep 19 08:59:53.237666 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.237645    2566 scope.go:117] "RemoveContainer" containerID="6b01ff38369e150ede0419266b8168541a90a27eb4c13b02213f5f73d8b98f2d"
Sep 19 08:59:53.244673 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.244654    2566 scope.go:117] "RemoveContainer" containerID="8eb01833134b75df05f38aa01e380563759b2a61e99995c5c294dd491938287b"
Sep 19 08:59:53.266076 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.266048    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:59:53.268459 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:53.268438    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-fv7p6"]
Sep 19 08:59:54.181672 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.181638    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" path="/var/lib/kubelet/pods/e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f/volumes"
Sep 19 08:59:54.237750 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.237713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerStarted","Data":"f901b7c8788daaa33c8c2f73a836caa315a9c18d8d73cd2477c13444fe665f30"}
Sep 19 08:59:54.238246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.237826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:59:54.238246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.237872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerStarted","Data":"2960ca817e2cd0ac6be0aa5fe8a1abddeb9491864908e8bbc4c0ffa217a3f89f"}
Sep 19 08:59:54.238246 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.238088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 08:59:54.239374 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.239349    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:59:54.256222 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:54.256181    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podStartSLOduration=2.351023611 podStartE2EDuration="1m58.256167297s" totalImagesPullingTime="1m51.946447749s" totalInitContainerRuntime="3.958695937s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:59:54.254996263 +0000 UTC m=+2358.575667852" watchObservedRunningTime="2026-09-19 08:59:54.256167297 +0000 UTC m=+2358.576838885"
Sep 19 08:59:55.241585 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:55.241552    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:59:58.255640 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:58.255605    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a68574-29c5-40bc-af35-40ef669a77b6" containerID="bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf" exitCode=0
Sep 19 08:59:58.256053 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:58.255663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerDied","Data":"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"}
Sep 19 08:59:59.261312 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:59.261270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerStarted","Data":"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"}
Sep 19 08:59:59.261805 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:59.261340    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerStarted","Data":"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"}
Sep 19 08:59:59.261805 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:59.261562    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 08:59:59.282613 ip-10-0-136-228 kubenswrapper[2566]: I0919 08:59:59.282564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podStartSLOduration=25.20271835 podStartE2EDuration="29.282548456s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079830106s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:59:59.282054706 +0000 UTC m=+2363.602726297" watchObservedRunningTime="2026-09-19 08:59:59.282548456 +0000 UTC m=+2363.603220044"
Sep 19 09:00:00.246647 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:00.246618    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 09:00:00.247471 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:00.247451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 09:00:00.265787 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:00.265742    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 09:00:00.266761 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:00.266735    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:01.269004 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:01.268960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:06.273196 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:06.273170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 09:00:06.273633 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:06.273604    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:08.759064 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:08.759026    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 09:00:08.759523 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:08.759292    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container" containerID="cri-o://2960ca817e2cd0ac6be0aa5fe8a1abddeb9491864908e8bbc4c0ffa217a3f89f" gracePeriod=30
Sep 19 09:00:08.759523 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:08.759326    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kube-rbac-proxy" containerID="cri-o://f901b7c8788daaa33c8c2f73a836caa315a9c18d8d73cd2477c13444fe665f30" gracePeriod=30
Sep 19 09:00:08.769366 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:08.769339    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:00:09.301537 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:09.301503    2566 generic.go:356] "Generic (PLEG): container finished" podID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerID="f901b7c8788daaa33c8c2f73a836caa315a9c18d8d73cd2477c13444fe665f30" exitCode=2
Sep 19 09:00:09.301717 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:09.301575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerDied","Data":"f901b7c8788daaa33c8c2f73a836caa315a9c18d8d73cd2477c13444fe665f30"}
Sep 19 09:00:10.242351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:10.242314    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 19 09:00:10.247688 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:10.247658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 09:00:11.311639 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.311590    2566 generic.go:356] "Generic (PLEG): container finished" podID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerID="2960ca817e2cd0ac6be0aa5fe8a1abddeb9491864908e8bbc4c0ffa217a3f89f" exitCode=0
Sep 19 09:00:11.311977 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.311647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerDied","Data":"2960ca817e2cd0ac6be0aa5fe8a1abddeb9491864908e8bbc4c0ffa217a3f89f"}
Sep 19 09:00:11.433992 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.433969    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 09:00:11.537166 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") pod \"738fb929-79c1-4868-8b52-380fd2d62a8d\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") "
Sep 19 09:00:11.537166 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537154    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnk4t\" (UniqueName: \"kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t\") pod \"738fb929-79c1-4868-8b52-380fd2d62a8d\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") "
Sep 19 09:00:11.537374 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location\") pod \"738fb929-79c1-4868-8b52-380fd2d62a8d\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") "
Sep 19 09:00:11.537374 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config\") pod \"738fb929-79c1-4868-8b52-380fd2d62a8d\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") "
Sep 19 09:00:11.537487 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert\") pod \"738fb929-79c1-4868-8b52-380fd2d62a8d\" (UID: \"738fb929-79c1-4868-8b52-380fd2d62a8d\") "
Sep 19 09:00:11.537683 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "738fb929-79c1-4868-8b52-380fd2d62a8d" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:11.537781 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "738fb929-79c1-4868-8b52-380fd2d62a8d" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:00:11.537849 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.537812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "738fb929-79c1-4868-8b52-380fd2d62a8d" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:00:11.539268 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.539241    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t" (OuterVolumeSpecName: "kube-api-access-pnk4t") pod "738fb929-79c1-4868-8b52-380fd2d62a8d" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d"). InnerVolumeSpecName "kube-api-access-pnk4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:00:11.539368 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.539282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "738fb929-79c1-4868-8b52-380fd2d62a8d" (UID: "738fb929-79c1-4868-8b52-380fd2d62a8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:00:11.638097 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.638059    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/738fb929-79c1-4868-8b52-380fd2d62a8d-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:00:11.638097 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.638085    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnk4t\" (UniqueName: \"kubernetes.io/projected/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-api-access-pnk4t\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:00:11.638097 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.638095    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/738fb929-79c1-4868-8b52-380fd2d62a8d-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:00:11.638097 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.638104    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:00:11.638406 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:11.638114    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/738fb929-79c1-4868-8b52-380fd2d62a8d-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:00:12.317764 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.317731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt" event={"ID":"738fb929-79c1-4868-8b52-380fd2d62a8d","Type":"ContainerDied","Data":"52b66d06cd7522d24bf00926d799689a82d29ac55d98f333537ac77a3ebce44d"}
Sep 19 09:00:12.318198 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.317811    2566 scope.go:117] "RemoveContainer" containerID="f901b7c8788daaa33c8c2f73a836caa315a9c18d8d73cd2477c13444fe665f30"
Sep 19 09:00:12.318198 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.317742    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"
Sep 19 09:00:12.327479 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.327461    2566 scope.go:117] "RemoveContainer" containerID="2960ca817e2cd0ac6be0aa5fe8a1abddeb9491864908e8bbc4c0ffa217a3f89f"
Sep 19 09:00:12.335057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.335039    2566 scope.go:117] "RemoveContainer" containerID="d9c9156d6ed83ec3bc7083e299f7a5a5a17f48420e08a611df89d1eec83dcb8d"
Sep 19 09:00:12.337396 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.337374    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 09:00:12.341002 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:12.340980    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-hg4zt"]
Sep 19 09:00:13.942445 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.942405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:00:13.982855 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.982823    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:00:13.983178 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983162    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983181    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983195    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983204    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983215    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983225    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983241    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983248    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983265    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="storage-initializer"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983273    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="storage-initializer"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983288    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="storage-initializer"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="storage-initializer"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983371    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983382    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" containerName="kserve-container"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983392    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kube-rbac-proxy"
Sep 19 09:00:13.983613 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:13.983403    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7dd54f2-2f1b-4de4-9be2-fca6f237ab2f" containerName="kserve-container"
Sep 19 09:00:14.042573 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.042531    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:00:14.042708 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.042640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.045521 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.045498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 19 09:00:14.045884 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.045866    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 19 09:00:14.157595 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.157573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.157709 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.157605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.157709 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.157632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.157833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.157727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5tb\" (UniqueName: \"kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.157833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.157763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.165367 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.165340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:00:14.180295 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.180263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738fb929-79c1-4868-8b52-380fd2d62a8d" path="/var/lib/kubelet/pods/738fb929-79c1-4868-8b52-380fd2d62a8d/volumes"
Sep 19 09:00:14.258419 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.258419 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.258569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.258569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5tb\" (UniqueName: \"kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.258569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.258914 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.258892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.259077 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.259059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.259128 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.259111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.260852 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.260832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.268751 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.268732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5tb\" (UniqueName: \"kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb\") pod \"isvc-xgboost-predictor-7566d65bf7-2kbs6\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.353062 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.353036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:14.643146 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:14.643121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:00:14.644734 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:00:14.644692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61d170f_beae_4fa3_944c_17887659a3b9.slice/crio-18ce720ba83f82d2d1dab85214568f62272189785c50434b4250c34c2deb3957 WatchSource:0}: Error finding container 18ce720ba83f82d2d1dab85214568f62272189785c50434b4250c34c2deb3957: Status 404 returned error can't find the container with id 18ce720ba83f82d2d1dab85214568f62272189785c50434b4250c34c2deb3957
Sep 19 09:00:15.336499 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:15.336459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerStarted","Data":"508cc468baf9ff6eec8215feae8e0840c137f1f8126b10f6beea200b833baa33"}
Sep 19 09:00:15.336893 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:15.336534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerStarted","Data":"18ce720ba83f82d2d1dab85214568f62272189785c50434b4250c34c2deb3957"}
Sep 19 09:00:16.274235 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:16.274201    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:19.354174 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:19.354139    2566 generic.go:356] "Generic (PLEG): container finished" podID="d61d170f-beae-4fa3-944c-17887659a3b9" containerID="508cc468baf9ff6eec8215feae8e0840c137f1f8126b10f6beea200b833baa33" exitCode=0
Sep 19 09:00:19.354565 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:19.354212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerDied","Data":"508cc468baf9ff6eec8215feae8e0840c137f1f8126b10f6beea200b833baa33"}
Sep 19 09:00:26.273901 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:26.273744    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:36.274654 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:36.274615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:39.446316 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:39.446278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerStarted","Data":"59ecafafa81a2442ee440f0709d7638ca6900e1dfdd29b828591b6dd45c6acdd"}
Sep 19 09:00:40.452519 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:40.452472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerStarted","Data":"50f9428b55fbbcf378fec351295f68abc45a491912a61458c943d2da5136eb82"}
Sep 19 09:00:40.452940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:40.452761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:40.452940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:40.452827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:40.454257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:40.454227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:00:40.471640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:40.471600    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podStartSLOduration=3.56028357 podStartE2EDuration="27.471587591s" totalImagesPullingTime="19.99172178s" totalInitContainerRuntime="3.919582241s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:00:40.47011592 +0000 UTC m=+2404.790787509" watchObservedRunningTime="2026-09-19 09:00:40.471587591 +0000 UTC m=+2404.792259179"
Sep 19 09:00:41.457062 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:41.457027    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:00:46.273931 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:46.273893    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:46.461304 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:46.461275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:00:46.461841 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:46.461813    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:00:56.273578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:56.273540    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:00:56.462231 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:00:56.462191    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:06.274355 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:06.274275    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:01:06.462027 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:06.461990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:16.273762 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:16.273723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 09:01:16.461719 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:16.461682    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:18.180988 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:18.180962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 09:01:20.143165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.143124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:20.171575 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.171540    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 09:01:20.171821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.171753    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container" containerID="cri-o://31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52" gracePeriod=30
Sep 19 09:01:20.171821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.171807    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kube-rbac-proxy" containerID="cri-o://8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326" gracePeriod=30
Sep 19 09:01:20.202324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.202292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:20.610577 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.610551    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a68574-29c5-40bc-af35-40ef669a77b6" containerID="8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326" exitCode=2
Sep 19 09:01:20.610732 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:20.610589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerDied","Data":"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"}
Sep 19 09:01:21.269038 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:21.268995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 19 09:01:24.804522 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.804495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 09:01:24.927858 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.927751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config\") pod \"21a68574-29c5-40bc-af35-40ef669a77b6\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") "
Sep 19 09:01:24.927858 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.927809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") pod \"21a68574-29c5-40bc-af35-40ef669a77b6\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") "
Sep 19 09:01:24.927858 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.927850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert\") pod \"21a68574-29c5-40bc-af35-40ef669a77b6\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") "
Sep 19 09:01:24.928127 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.927894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location\") pod \"21a68574-29c5-40bc-af35-40ef669a77b6\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") "
Sep 19 09:01:24.928127 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.927919    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j57nd\" (UniqueName: \"kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd\") pod \"21a68574-29c5-40bc-af35-40ef669a77b6\" (UID: \"21a68574-29c5-40bc-af35-40ef669a77b6\") "
Sep 19 09:01:24.928283 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.928258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21a68574-29c5-40bc-af35-40ef669a77b6" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:24.928352 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.928259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21a68574-29c5-40bc-af35-40ef669a77b6" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:01:24.928352 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.928322    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21a68574-29c5-40bc-af35-40ef669a77b6" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:24.930094 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.930067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21a68574-29c5-40bc-af35-40ef669a77b6" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:01:24.930188 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:24.930142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd" (OuterVolumeSpecName: "kube-api-access-j57nd") pod "21a68574-29c5-40bc-af35-40ef669a77b6" (UID: "21a68574-29c5-40bc-af35-40ef669a77b6"). InnerVolumeSpecName "kube-api-access-j57nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:01:25.028961 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.028931    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21a68574-29c5-40bc-af35-40ef669a77b6-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:25.028961 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.028956    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j57nd\" (UniqueName: \"kubernetes.io/projected/21a68574-29c5-40bc-af35-40ef669a77b6-kube-api-access-j57nd\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:25.028961 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.028966    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:25.029175 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.028976    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21a68574-29c5-40bc-af35-40ef669a77b6-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:25.029175 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.028986    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21a68574-29c5-40bc-af35-40ef669a77b6-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:25.635923 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.635892    2566 generic.go:356] "Generic (PLEG): container finished" podID="21a68574-29c5-40bc-af35-40ef669a77b6" containerID="31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52" exitCode=0
Sep 19 09:01:25.636098 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.635948    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"
Sep 19 09:01:25.636098 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.635981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerDied","Data":"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"}
Sep 19 09:01:25.636098 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.636039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9" event={"ID":"21a68574-29c5-40bc-af35-40ef669a77b6","Type":"ContainerDied","Data":"68113984bc139dfc7aa605690a0fafda505cedb03a03fc4f84560c7c60cb9c70"}
Sep 19 09:01:25.636098 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.636066    2566 scope.go:117] "RemoveContainer" containerID="8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"
Sep 19 09:01:25.644314 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.644294    2566 scope.go:117] "RemoveContainer" containerID="31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"
Sep 19 09:01:25.651414 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.651398    2566 scope.go:117] "RemoveContainer" containerID="bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"
Sep 19 09:01:25.658528 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.658509    2566 scope.go:117] "RemoveContainer" containerID="8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"
Sep 19 09:01:25.658596 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.658510    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 09:01:25.658826 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:01:25.658805    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326\": container with ID starting with 8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326 not found: ID does not exist" containerID="8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"
Sep 19 09:01:25.658885 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.658833    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326"} err="failed to get container status \"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326\": rpc error: code = NotFound desc = could not find container \"8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326\": container with ID starting with 8319f72918127b192097dca0702e0c184c3b1b678314bf6201d9d8a9c2d61326 not found: ID does not exist"
Sep 19 09:01:25.658885 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.658848    2566 scope.go:117] "RemoveContainer" containerID="31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"
Sep 19 09:01:25.659101 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:01:25.659084    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52\": container with ID starting with 31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52 not found: ID does not exist" containerID="31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"
Sep 19 09:01:25.659164 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.659110    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52"} err="failed to get container status \"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52\": rpc error: code = NotFound desc = could not find container \"31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52\": container with ID starting with 31f1895a704d81143070452819955a4986b6dc1437932f2bd9b2aad04010ea52 not found: ID does not exist"
Sep 19 09:01:25.659164 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.659133    2566 scope.go:117] "RemoveContainer" containerID="bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"
Sep 19 09:01:25.659369 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:01:25.659353    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf\": container with ID starting with bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf not found: ID does not exist" containerID="bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"
Sep 19 09:01:25.659415 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.659372    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf"} err="failed to get container status \"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf\": rpc error: code = NotFound desc = could not find container \"bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf\": container with ID starting with bf72f1ccb662dbe98ff26b0ee188811bb9b704d670c7b3bbc44e9c70246bc0cf not found: ID does not exist"
Sep 19 09:01:25.663652 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:25.663632    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wx4j9"]
Sep 19 09:01:26.180818 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:26.180762    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" path="/var/lib/kubelet/pods/21a68574-29c5-40bc-af35-40ef669a77b6/volumes"
Sep 19 09:01:26.462155 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:26.462070    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:30.439686 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.439649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:30.470627 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470598    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:01:30.470887 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kube-rbac-proxy"
Sep 19 09:01:30.470934 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470889    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kube-rbac-proxy"
Sep 19 09:01:30.470934 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470901    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container"
Sep 19 09:01:30.470934 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470906    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container"
Sep 19 09:01:30.470934 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470918    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="storage-initializer"
Sep 19 09:01:30.470934 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470924    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="storage-initializer"
Sep 19 09:01:30.471088 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470967    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kube-rbac-proxy"
Sep 19 09:01:30.471088 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.470974    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a68574-29c5-40bc-af35-40ef669a77b6" containerName="kserve-container"
Sep 19 09:01:30.473567 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.473551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.478756 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.478736    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 19 09:01:30.479055 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.479031    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 19 09:01:30.486753 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.486731    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:01:30.556215 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.556185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:30.573234 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.573207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffql\" (UniqueName: \"kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.573342 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.573242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.573342 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.573265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.573342 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.573324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.573444 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.573364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.674584 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.674554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.674746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.674599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.674746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.674634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffql\" (UniqueName: \"kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.674746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.674664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.674746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.674692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.675204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.675182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.675461 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.675441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.675497 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.675441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.677314 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.677298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.683114 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.683092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffql\" (UniqueName: \"kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql\") pod \"isvc-lightgbm-predictor-54f85ddf88-xwhpf\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:30.783346 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:30.783259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:31.056035 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:31.056011    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:01:31.058318 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:01:31.058289    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37d86de_4793_4289_9602_463750dedd0e.slice/crio-f368d6b6baa10590d20ddd5b563ec3f301fcf44ed4f04c9e3df5005cd9eede78 WatchSource:0}: Error finding container f368d6b6baa10590d20ddd5b563ec3f301fcf44ed4f04c9e3df5005cd9eede78: Status 404 returned error can't find the container with id f368d6b6baa10590d20ddd5b563ec3f301fcf44ed4f04c9e3df5005cd9eede78
Sep 19 09:01:31.661939 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:31.661906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerStarted","Data":"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"}
Sep 19 09:01:31.662314 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:31.661962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerStarted","Data":"f368d6b6baa10590d20ddd5b563ec3f301fcf44ed4f04c9e3df5005cd9eede78"}
Sep 19 09:01:35.678942 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:35.678908    2566 generic.go:356] "Generic (PLEG): container finished" podID="a37d86de-4793-4289-9602-463750dedd0e" containerID="6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a" exitCode=0
Sep 19 09:01:35.679320 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:35.678967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerDied","Data":"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"}
Sep 19 09:01:35.680322 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:35.680304    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:01:36.461830 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:36.461789    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:42.712114 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.712081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerStarted","Data":"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"}
Sep 19 09:01:42.712535 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.712145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerStarted","Data":"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"}
Sep 19 09:01:42.712535 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.712416    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:42.712634 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.712599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:42.713906 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.713877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:01:42.733967 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:42.733923    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podStartSLOduration=2.534939816 podStartE2EDuration="12.733910397s" totalImagesPullingTime="6.309032909s" totalInitContainerRuntime="3.889937672s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:01:42.731528097 +0000 UTC m=+2467.052199684" watchObservedRunningTime="2026-09-19 09:01:42.733910397 +0000 UTC m=+2467.054581984"
Sep 19 09:01:43.716828 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:43.716787    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:01:46.462924 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:46.462892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:01:48.721884 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:48.721855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:01:48.722326 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:48.722302    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:01:54.460429 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.460394    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:01:54.460894 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.460627    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" containerID="cri-o://59ecafafa81a2442ee440f0709d7638ca6900e1dfdd29b828591b6dd45c6acdd" gracePeriod=30
Sep 19 09:01:54.460894 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.460692    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kube-rbac-proxy" containerID="cri-o://50f9428b55fbbcf378fec351295f68abc45a491912a61458c943d2da5136eb82" gracePeriod=30
Sep 19 09:01:54.481467 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.481442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:54.761976 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.761892    2566 generic.go:356] "Generic (PLEG): container finished" podID="d61d170f-beae-4fa3-944c-17887659a3b9" containerID="50f9428b55fbbcf378fec351295f68abc45a491912a61458c943d2da5136eb82" exitCode=2
Sep 19 09:01:54.761976 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:54.761960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerDied","Data":"50f9428b55fbbcf378fec351295f68abc45a491912a61458c943d2da5136eb82"}
Sep 19 09:01:56.457922 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:56.457879    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 19 09:01:56.462352 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:56.462326    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 09:01:58.723183 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.723143    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:01:58.779274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.779238    2566 generic.go:356] "Generic (PLEG): container finished" podID="d61d170f-beae-4fa3-944c-17887659a3b9" containerID="59ecafafa81a2442ee440f0709d7638ca6900e1dfdd29b828591b6dd45c6acdd" exitCode=0
Sep 19 09:01:58.779440 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.779311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerDied","Data":"59ecafafa81a2442ee440f0709d7638ca6900e1dfdd29b828591b6dd45c6acdd"}
Sep 19 09:01:58.805156 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.805131    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:01:58.921451 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config\") pod \"d61d170f-beae-4fa3-944c-17887659a3b9\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") "
Sep 19 09:01:58.921451 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921398    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location\") pod \"d61d170f-beae-4fa3-944c-17887659a3b9\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") "
Sep 19 09:01:58.921451 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921447    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls\") pod \"d61d170f-beae-4fa3-944c-17887659a3b9\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") "
Sep 19 09:01:58.921694 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert\") pod \"d61d170f-beae-4fa3-944c-17887659a3b9\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") "
Sep 19 09:01:58.921694 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k5tb\" (UniqueName: \"kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb\") pod \"d61d170f-beae-4fa3-944c-17887659a3b9\" (UID: \"d61d170f-beae-4fa3-944c-17887659a3b9\") "
Sep 19 09:01:58.921831 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d61d170f-beae-4fa3-944c-17887659a3b9" (UID: "d61d170f-beae-4fa3-944c-17887659a3b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:58.921908 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d61d170f-beae-4fa3-944c-17887659a3b9" (UID: "d61d170f-beae-4fa3-944c-17887659a3b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:01:58.921965 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921922    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d61d170f-beae-4fa3-944c-17887659a3b9" (UID: "d61d170f-beae-4fa3-944c-17887659a3b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:58.921965 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921945    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:58.921965 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.921961    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d61d170f-beae-4fa3-944c-17887659a3b9-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:58.923572 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.923548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb" (OuterVolumeSpecName: "kube-api-access-6k5tb") pod "d61d170f-beae-4fa3-944c-17887659a3b9" (UID: "d61d170f-beae-4fa3-944c-17887659a3b9"). InnerVolumeSpecName "kube-api-access-6k5tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:01:58.923644 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:58.923589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d61d170f-beae-4fa3-944c-17887659a3b9" (UID: "d61d170f-beae-4fa3-944c-17887659a3b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:01:59.022403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.022372    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d61d170f-beae-4fa3-944c-17887659a3b9-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:59.022403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.022397    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5tb\" (UniqueName: \"kubernetes.io/projected/d61d170f-beae-4fa3-944c-17887659a3b9-kube-api-access-6k5tb\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:59.022403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.022406    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d61d170f-beae-4fa3-944c-17887659a3b9-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:01:59.632711 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.632677    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:01:59.662511 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662478    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:01:59.662756 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662744    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container"
Sep 19 09:01:59.662819 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662758    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container"
Sep 19 09:01:59.662819 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662780    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kube-rbac-proxy"
Sep 19 09:01:59.662819 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662786    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kube-rbac-proxy"
Sep 19 09:01:59.662819 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="storage-initializer"
Sep 19 09:01:59.662819 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662808    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="storage-initializer"
Sep 19 09:01:59.662986 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662857    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kserve-container"
Sep 19 09:01:59.662986 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.662864    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" containerName="kube-rbac-proxy"
Sep 19 09:01:59.666129 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.666112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.668499 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.668474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 19 09:01:59.668499 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.668491    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 19 09:01:59.675490 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.675467    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:01:59.785697 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.785669    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"
Sep 19 09:01:59.786194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.785669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6" event={"ID":"d61d170f-beae-4fa3-944c-17887659a3b9","Type":"ContainerDied","Data":"18ce720ba83f82d2d1dab85214568f62272189785c50434b4250c34c2deb3957"}
Sep 19 09:01:59.786194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.785829    2566 scope.go:117] "RemoveContainer" containerID="50f9428b55fbbcf378fec351295f68abc45a491912a61458c943d2da5136eb82"
Sep 19 09:01:59.794310 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.794290    2566 scope.go:117] "RemoveContainer" containerID="59ecafafa81a2442ee440f0709d7638ca6900e1dfdd29b828591b6dd45c6acdd"
Sep 19 09:01:59.801858 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.801838    2566 scope.go:117] "RemoveContainer" containerID="508cc468baf9ff6eec8215feae8e0840c137f1f8126b10f6beea200b833baa33"
Sep 19 09:01:59.808313 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.808285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:01:59.811877 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.811853    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-2kbs6"]
Sep 19 09:01:59.830597 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.830570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.830725 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.830619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.830725 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.830647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.830829 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.830723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5dn\" (UniqueName: \"kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.830829 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.830789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932142 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5dn\" (UniqueName: \"kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932142 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932354 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932354 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:01:59.932334    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 19 09:01:59.932449 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:01:59.932397    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls podName:6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4 nodeName:}" failed. No retries permitted until 2026-09-19 09:02:00.432377164 +0000 UTC m=+2484.753048742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 19 09:01:59.932643 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932859 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.932903 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.932845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:01:59.941098 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:01:59.941072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5dn\" (UniqueName: \"kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:00.180975 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.180939    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d61d170f-beae-4fa3-944c-17887659a3b9" path="/var/lib/kubelet/pods/d61d170f-beae-4fa3-944c-17887659a3b9/volumes"
Sep 19 09:02:00.288139 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.288057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:00.436712 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.436670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:00.439153 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.439113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:00.576451 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.576367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:00.849057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:00.848657    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:02:00.851571 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:02:00.851543    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6058d521_5692_4f4b_b4f7_2d7bfb7dbdb4.slice/crio-214da170c6339e235442cd8205d85853046e6978cf3d3b792d7f6c381cfbfce5 WatchSource:0}: Error finding container 214da170c6339e235442cd8205d85853046e6978cf3d3b792d7f6c381cfbfce5: Status 404 returned error can't find the container with id 214da170c6339e235442cd8205d85853046e6978cf3d3b792d7f6c381cfbfce5
Sep 19 09:02:01.797671 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:01.797635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerStarted","Data":"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"}
Sep 19 09:02:01.797845 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:01.797694    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerStarted","Data":"214da170c6339e235442cd8205d85853046e6978cf3d3b792d7f6c381cfbfce5"}
Sep 19 09:02:05.815221 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:05.815190    2566 generic.go:356] "Generic (PLEG): container finished" podID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerID="85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74" exitCode=0
Sep 19 09:02:05.815596 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:05.815265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerDied","Data":"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"}
Sep 19 09:02:06.821195 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:06.821162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerStarted","Data":"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"}
Sep 19 09:02:06.821566 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:06.821250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerStarted","Data":"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"}
Sep 19 09:02:06.821566 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:06.821448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:07.824844 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:07.824811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:08.723324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:08.723287    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:13.834158 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:13.834128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:13.858147 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:13.858102    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" podStartSLOduration=10.934340228 podStartE2EDuration="14.858082064s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.923741836s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:02:06.84613524 +0000 UTC m=+2491.166806840" watchObservedRunningTime="2026-09-19 09:02:13.858082064 +0000 UTC m=+2498.178753653"
Sep 19 09:02:18.723276 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:18.723220    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:28.723046 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:28.722967    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:38.722999 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:38.722950    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:43.844757 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:43.844725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:45.626172 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:45.626127    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:48.722554 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:48.722518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:49.707325 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.707282    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:02:49.707943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.707908    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kserve-container" containerID="cri-o://0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d" gracePeriod=30
Sep 19 09:02:49.708374 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.708318    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kube-rbac-proxy" containerID="cri-o://633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9" gracePeriod=30
Sep 19 09:02:49.727422 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.727396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:49.989753 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.989673    2566 generic.go:356] "Generic (PLEG): container finished" podID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerID="633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9" exitCode=2
Sep 19 09:02:49.989753 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:49.989732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerDied","Data":"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"}
Sep 19 09:02:50.176097 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:50.176057    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:02:53.828959 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:53.828917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 09:02:53.835449 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:53.835415    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 09:02:54.880802 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.880753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:54.908150 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.908114    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:02:54.910446 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.910427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:54.913389 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.913360    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 19 09:02:54.913389 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.913372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 19 09:02:54.927324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.927296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:02:54.954274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.954242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:54.954413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.954283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:54.954413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.954306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:54.954413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.954389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7b7\" (UniqueName: \"kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:54.954526 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:54.954424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.026953 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.026927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:55.054946 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.054922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.054956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.054981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7b7\" (UniqueName: \"kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.054999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.055055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055513 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.055477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055624 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.055607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.055712 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.055694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.057415 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.057387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.063184 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.063164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7b7\" (UniqueName: \"kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.220631 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.220546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:02:55.492862 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.492832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:02:55.495503 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:02:55.495472    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5c733d_34a5_487c_a742_3195fb220ff8.slice/crio-c18845e8d28aef02a30c8a9637840ea49c0aaf8d24dae27c6d3b4d848121082d WatchSource:0}: Error finding container c18845e8d28aef02a30c8a9637840ea49c0aaf8d24dae27c6d3b4d848121082d: Status 404 returned error can't find the container with id c18845e8d28aef02a30c8a9637840ea49c0aaf8d24dae27c6d3b4d848121082d
Sep 19 09:02:55.536220 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.536198    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:55.579134 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.579097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:02:55.660203 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert\") pod \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") "
Sep 19 09:02:55.660380 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location\") pod \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") "
Sep 19 09:02:55.660380 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config\") pod \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") "
Sep 19 09:02:55.660380 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5dn\" (UniqueName: \"kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn\") pod \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") "
Sep 19 09:02:55.660619 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") pod \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\" (UID: \"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4\") "
Sep 19 09:02:55.660619 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660582    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:55.660716 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:02:55.660716 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.660688    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:02:55.662466 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.662433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:02:55.662466 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.662447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn" (OuterVolumeSpecName: "kube-api-access-7n5dn") pod "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" (UID: "6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4"). InnerVolumeSpecName "kube-api-access-7n5dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:02:55.761574 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.761486    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:02:55.761574 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.761516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:02:55.761574 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.761530    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5dn\" (UniqueName: \"kubernetes.io/projected/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-kube-api-access-7n5dn\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:02:55.761574 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.761543    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:02:55.761574 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:55.761556    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:02:56.020421 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.020335    2566 generic.go:356] "Generic (PLEG): container finished" podID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerID="0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d" exitCode=0
Sep 19 09:02:56.020864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.020449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerDied","Data":"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"}
Sep 19 09:02:56.020864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.020463    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"
Sep 19 09:02:56.020864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.020549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh" event={"ID":"6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4","Type":"ContainerDied","Data":"214da170c6339e235442cd8205d85853046e6978cf3d3b792d7f6c381cfbfce5"}
Sep 19 09:02:56.020864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.020590    2566 scope.go:117] "RemoveContainer" containerID="633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"
Sep 19 09:02:56.022365 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.022343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerStarted","Data":"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"}
Sep 19 09:02:56.022476 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.022425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerStarted","Data":"c18845e8d28aef02a30c8a9637840ea49c0aaf8d24dae27c6d3b4d848121082d"}
Sep 19 09:02:56.030305 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.030288    2566 scope.go:117] "RemoveContainer" containerID="0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"
Sep 19 09:02:56.037298 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.037282    2566 scope.go:117] "RemoveContainer" containerID="85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"
Sep 19 09:02:56.045845 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.045826    2566 scope.go:117] "RemoveContainer" containerID="633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"
Sep 19 09:02:56.046285 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:02:56.046257    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9\": container with ID starting with 633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9 not found: ID does not exist" containerID="633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"
Sep 19 09:02:56.046357 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.046296    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9"} err="failed to get container status \"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9\": rpc error: code = NotFound desc = could not find container \"633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9\": container with ID starting with 633a2c8c875f6b8e484e4b99b39e8e3b03f1fcd0a717bfc972d8c59a1bda65d9 not found: ID does not exist"
Sep 19 09:02:56.046357 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.046320    2566 scope.go:117] "RemoveContainer" containerID="0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"
Sep 19 09:02:56.046733 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:02:56.046710    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d\": container with ID starting with 0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d not found: ID does not exist" containerID="0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"
Sep 19 09:02:56.046824 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.046738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d"} err="failed to get container status \"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d\": rpc error: code = NotFound desc = could not find container \"0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d\": container with ID starting with 0aa898df838fa25df4035f35f3da23536789564818613b9b6a1f784779353f4d not found: ID does not exist"
Sep 19 09:02:56.046824 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.046754    2566 scope.go:117] "RemoveContainer" containerID="85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"
Sep 19 09:02:56.047086 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:02:56.047068    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74\": container with ID starting with 85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74 not found: ID does not exist" containerID="85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"
Sep 19 09:02:56.047138 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.047091    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74"} err="failed to get container status \"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74\": rpc error: code = NotFound desc = could not find container \"85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74\": container with ID starting with 85aa64764dfe9f96f3230483dd4d7adfc2d4f1599102bae36b253658b677ae74 not found: ID does not exist"
Sep 19 09:02:56.057518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.057494    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:02:56.060763 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.060736    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-4g4fh"]
Sep 19 09:02:56.180414 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:02:56.180379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" path="/var/lib/kubelet/pods/6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4/volumes"
Sep 19 09:03:00.040501 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:00.040419    2566 generic.go:356] "Generic (PLEG): container finished" podID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerID="e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59" exitCode=0
Sep 19 09:03:00.040501 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:00.040493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerDied","Data":"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"}
Sep 19 09:03:00.176237 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:00.176206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:03:01.047405 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:01.047360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerStarted","Data":"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"}
Sep 19 09:03:01.047815 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:01.047477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerStarted","Data":"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"}
Sep 19 09:03:01.047815 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:01.047715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:01.069831 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:01.069784    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" podStartSLOduration=2.98542225 podStartE2EDuration="7.069752959s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.084330709s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:03:01.068369214 +0000 UTC m=+2545.389040815" watchObservedRunningTime="2026-09-19 09:03:01.069752959 +0000 UTC m=+2545.390424547"
Sep 19 09:03:02.051949 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:02.051907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:08.061280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:08.061251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:10.176744 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:10.176705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 19 09:03:18.478003 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:18.477963    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:20.180586 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:20.180554    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:03:21.459115 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:21.459082    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:03:21.459505 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:21.459367    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container" containerID="cri-o://a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3" gracePeriod=30
Sep 19 09:03:21.459505 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:21.459407    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kube-rbac-proxy" containerID="cri-o://02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1" gracePeriod=30
Sep 19 09:03:21.474783 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:21.474749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:22.130419 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:22.130384    2566 generic.go:356] "Generic (PLEG): container finished" podID="a37d86de-4793-4289-9602-463750dedd0e" containerID="02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1" exitCode=2
Sep 19 09:03:22.130587 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:22.130455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerDied","Data":"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"}
Sep 19 09:03:23.717608 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:23.717567    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 19 09:03:26.672955 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:26.672858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:26.991620 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:26.991600    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:03:27.102978 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.102944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert\") pod \"a37d86de-4793-4289-9602-463750dedd0e\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") "
Sep 19 09:03:27.103157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.102989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config\") pod \"a37d86de-4793-4289-9602-463750dedd0e\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") "
Sep 19 09:03:27.103157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103046    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hffql\" (UniqueName: \"kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql\") pod \"a37d86de-4793-4289-9602-463750dedd0e\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") "
Sep 19 09:03:27.103157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location\") pod \"a37d86de-4793-4289-9602-463750dedd0e\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") "
Sep 19 09:03:27.103157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls\") pod \"a37d86de-4793-4289-9602-463750dedd0e\" (UID: \"a37d86de-4793-4289-9602-463750dedd0e\") "
Sep 19 09:03:27.103424 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103400    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a37d86de-4793-4289-9602-463750dedd0e" (UID: "a37d86de-4793-4289-9602-463750dedd0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:27.103424 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103406    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a37d86de-4793-4289-9602-463750dedd0e" (UID: "a37d86de-4793-4289-9602-463750dedd0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:03:27.103515 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.103479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a37d86de-4793-4289-9602-463750dedd0e" (UID: "a37d86de-4793-4289-9602-463750dedd0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:03:27.105187 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.105158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql" (OuterVolumeSpecName: "kube-api-access-hffql") pod "a37d86de-4793-4289-9602-463750dedd0e" (UID: "a37d86de-4793-4289-9602-463750dedd0e"). InnerVolumeSpecName "kube-api-access-hffql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:03:27.105187 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.105159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a37d86de-4793-4289-9602-463750dedd0e" (UID: "a37d86de-4793-4289-9602-463750dedd0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:03:27.150420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.150395    2566 generic.go:356] "Generic (PLEG): container finished" podID="a37d86de-4793-4289-9602-463750dedd0e" containerID="a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3" exitCode=0
Sep 19 09:03:27.150538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.150451    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"
Sep 19 09:03:27.150538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.150462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerDied","Data":"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"}
Sep 19 09:03:27.150538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.150510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf" event={"ID":"a37d86de-4793-4289-9602-463750dedd0e","Type":"ContainerDied","Data":"f368d6b6baa10590d20ddd5b563ec3f301fcf44ed4f04c9e3df5005cd9eede78"}
Sep 19 09:03:27.150538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.150534    2566 scope.go:117] "RemoveContainer" containerID="02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"
Sep 19 09:03:27.165294 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.165274    2566 scope.go:117] "RemoveContainer" containerID="a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"
Sep 19 09:03:27.172799 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.172781    2566 scope.go:117] "RemoveContainer" containerID="6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"
Sep 19 09:03:27.175396 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.175375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:03:27.179095 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.179075    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xwhpf"]
Sep 19 09:03:27.180444 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.180428    2566 scope.go:117] "RemoveContainer" containerID="02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"
Sep 19 09:03:27.180688 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:27.180672    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1\": container with ID starting with 02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1 not found: ID does not exist" containerID="02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"
Sep 19 09:03:27.180738 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.180696    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1"} err="failed to get container status \"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1\": rpc error: code = NotFound desc = could not find container \"02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1\": container with ID starting with 02ee907acaf32e226da72b608050d61849dbbddfe2d24cf6a767f577070f97b1 not found: ID does not exist"
Sep 19 09:03:27.180738 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.180711    2566 scope.go:117] "RemoveContainer" containerID="a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"
Sep 19 09:03:27.180954 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:27.180937    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3\": container with ID starting with a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3 not found: ID does not exist" containerID="a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"
Sep 19 09:03:27.180992 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.180961    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3"} err="failed to get container status \"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3\": rpc error: code = NotFound desc = could not find container \"a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3\": container with ID starting with a9487ca2685ea5e0ca7489da762e5894f53e598445d1398602e54b9633f05eb3 not found: ID does not exist"
Sep 19 09:03:27.180992 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.180980    2566 scope.go:117] "RemoveContainer" containerID="6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"
Sep 19 09:03:27.181237 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:27.181219    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a\": container with ID starting with 6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a not found: ID does not exist" containerID="6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"
Sep 19 09:03:27.181275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.181242    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a"} err="failed to get container status \"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a\": rpc error: code = NotFound desc = could not find container \"6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a\": container with ID starting with 6db42d9dfcdccac08cb8693d7ff3b2e70f982081c6f8c32eedb76cb477b7050a not found: ID does not exist"
Sep 19 09:03:27.203534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.203516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffql\" (UniqueName: \"kubernetes.io/projected/a37d86de-4793-4289-9602-463750dedd0e-kube-api-access-hffql\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:27.203603 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.203536    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a37d86de-4793-4289-9602-463750dedd0e-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:27.203603 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.203546    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a37d86de-4793-4289-9602-463750dedd0e-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:27.203603 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.203555    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:27.203603 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.203563    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a37d86de-4793-4289-9602-463750dedd0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:27.410481 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:27.410449    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:28.179832 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:28.179796    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37d86de-4793-4289-9602-463750dedd0e" path="/var/lib/kubelet/pods/a37d86de-4793-4289-9602-463750dedd0e/volumes"
Sep 19 09:03:38.065061 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:38.065032    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:44.944305 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:44.944274    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:03:44.944704 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:44.944522    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kserve-container" containerID="cri-o://613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3" gracePeriod=30
Sep 19 09:03:44.944704 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:44.944562    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kube-rbac-proxy" containerID="cri-o://2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d" gracePeriod=30
Sep 19 09:03:44.971047 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:44.971018    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:45.219003 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:45.218921    2566 generic.go:356] "Generic (PLEG): container finished" podID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerID="2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d" exitCode=2
Sep 19 09:03:45.219003 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:45.218981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerDied","Data":"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"}
Sep 19 09:03:48.057267 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:48.057223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 19 09:03:48.062560 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:48.062530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 09:03:50.110355 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.110325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:50.137881 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.137839    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:03:50.138123 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138111    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138125    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138135    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kserve-container"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138140    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kserve-container"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138148    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138154    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container"
Sep 19 09:03:50.138165 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138166    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138172    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138181    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="storage-initializer"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138186    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="storage-initializer"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138193    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="storage-initializer"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138200    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="storage-initializer"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138248    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kube-rbac-proxy"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138254    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6058d521-5692-4f4b-b4f7-2d7bfb7dbdb4" containerName="kserve-container"
Sep 19 09:03:50.138349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.138261    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37d86de-4793-4289-9602-463750dedd0e" containerName="kserve-container"
Sep 19 09:03:50.141310 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.141295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.143539 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.143510    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 19 09:03:50.143661 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.143545    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:03:50.150742 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.150719    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:03:50.176761 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.176738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.176881 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.176845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.176881 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.176865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.176987 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.176962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.177025 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.177012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtrj\" (UniqueName: \"kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.222316 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.222290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:03:50.277598 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.277569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.277598 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.277594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.277757 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.277619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.277757 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.277638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtrj\" (UniqueName: \"kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.277757 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.277662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.278105 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.278087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.278390 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.278372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.278466 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.278371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.279983 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.279965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.286477 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.286453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtrj\" (UniqueName: \"kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-98lf5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.452906 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.452832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:50.723968 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.723943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:03:50.728889 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:03:50.728857    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02c240a_0127_48ea_983c_c569e96272c5.slice/crio-8b2dc046210653fa15c25005e6bddc68362b79b1ef49e8b6566005906482f9a9 WatchSource:0}: Error finding container 8b2dc046210653fa15c25005e6bddc68362b79b1ef49e8b6566005906482f9a9: Status 404 returned error can't find the container with id 8b2dc046210653fa15c25005e6bddc68362b79b1ef49e8b6566005906482f9a9
Sep 19 09:03:50.772343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.772324    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:50.884243 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config\") pod \"7e5c733d-34a5-487c-a742-3195fb220ff8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") "
Sep 19 09:03:50.884420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884263    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location\") pod \"7e5c733d-34a5-487c-a742-3195fb220ff8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") "
Sep 19 09:03:50.884420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert\") pod \"7e5c733d-34a5-487c-a742-3195fb220ff8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") "
Sep 19 09:03:50.884420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf7b7\" (UniqueName: \"kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7\") pod \"7e5c733d-34a5-487c-a742-3195fb220ff8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") "
Sep 19 09:03:50.884420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls\") pod \"7e5c733d-34a5-487c-a742-3195fb220ff8\" (UID: \"7e5c733d-34a5-487c-a742-3195fb220ff8\") "
Sep 19 09:03:50.884651 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884623    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e5c733d-34a5-487c-a742-3195fb220ff8" (UID: "7e5c733d-34a5-487c-a742-3195fb220ff8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:03:50.884717 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884637    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e5c733d-34a5-487c-a742-3195fb220ff8" (UID: "7e5c733d-34a5-487c-a742-3195fb220ff8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:50.884763 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.884714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e5c733d-34a5-487c-a742-3195fb220ff8" (UID: "7e5c733d-34a5-487c-a742-3195fb220ff8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:03:50.886318 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.886293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7" (OuterVolumeSpecName: "kube-api-access-zf7b7") pod "7e5c733d-34a5-487c-a742-3195fb220ff8" (UID: "7e5c733d-34a5-487c-a742-3195fb220ff8"). InnerVolumeSpecName "kube-api-access-zf7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:03:50.886408 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.886347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e5c733d-34a5-487c-a742-3195fb220ff8" (UID: "7e5c733d-34a5-487c-a742-3195fb220ff8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:03:50.985383 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.985320    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:50.985383 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.985344    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e5c733d-34a5-487c-a742-3195fb220ff8-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:50.985383 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.985355    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e5c733d-34a5-487c-a742-3195fb220ff8-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:50.985383 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.985363    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7b7\" (UniqueName: \"kubernetes.io/projected/7e5c733d-34a5-487c-a742-3195fb220ff8-kube-api-access-zf7b7\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:50.985383 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:50.985373    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5c733d-34a5-487c-a742-3195fb220ff8-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:03:51.243106 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.243022    2566 generic.go:356] "Generic (PLEG): container finished" podID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerID="613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3" exitCode=0
Sep 19 09:03:51.243106 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.243093    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"
Sep 19 09:03:51.243599 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.243092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerDied","Data":"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"}
Sep 19 09:03:51.243599 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.243192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8" event={"ID":"7e5c733d-34a5-487c-a742-3195fb220ff8","Type":"ContainerDied","Data":"c18845e8d28aef02a30c8a9637840ea49c0aaf8d24dae27c6d3b4d848121082d"}
Sep 19 09:03:51.243599 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.243240    2566 scope.go:117] "RemoveContainer" containerID="2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"
Sep 19 09:03:51.244993 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.244967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerStarted","Data":"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"}
Sep 19 09:03:51.245122 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.245036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerStarted","Data":"8b2dc046210653fa15c25005e6bddc68362b79b1ef49e8b6566005906482f9a9"}
Sep 19 09:03:51.251975 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.251868    2566 scope.go:117] "RemoveContainer" containerID="613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"
Sep 19 09:03:51.258795 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.258764    2566 scope.go:117] "RemoveContainer" containerID="e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"
Sep 19 09:03:51.266222 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.266204    2566 scope.go:117] "RemoveContainer" containerID="2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"
Sep 19 09:03:51.266481 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:51.266453    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d\": container with ID starting with 2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d not found: ID does not exist" containerID="2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"
Sep 19 09:03:51.266570 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.266481    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d"} err="failed to get container status \"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d\": rpc error: code = NotFound desc = could not find container \"2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d\": container with ID starting with 2ae32b87fb868129d6727c861174cd99647cadb2aa40fcb5f4f7f13d42bee47d not found: ID does not exist"
Sep 19 09:03:51.266570 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.266499    2566 scope.go:117] "RemoveContainer" containerID="613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"
Sep 19 09:03:51.266762 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:51.266739    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3\": container with ID starting with 613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3 not found: ID does not exist" containerID="613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"
Sep 19 09:03:51.266849 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.266778    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3"} err="failed to get container status \"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3\": rpc error: code = NotFound desc = could not find container \"613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3\": container with ID starting with 613795583bc93e639dfb868f2295c6345dd1f1d3698b632e86fba745f5c0efd3 not found: ID does not exist"
Sep 19 09:03:51.266849 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.266798    2566 scope.go:117] "RemoveContainer" containerID="e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"
Sep 19 09:03:51.267041 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:03:51.267025    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59\": container with ID starting with e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59 not found: ID does not exist" containerID="e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"
Sep 19 09:03:51.267120 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.267045    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59"} err="failed to get container status \"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59\": rpc error: code = NotFound desc = could not find container \"e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59\": container with ID starting with e430d7f2f102353ac75511d7627104e775c6879742774b79cff369010199bd59 not found: ID does not exist"
Sep 19 09:03:51.278276 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.278248    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:03:51.283267 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:51.283245    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bgzv8"]
Sep 19 09:03:52.180398 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:52.180371    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" path="/var/lib/kubelet/pods/7e5c733d-34a5-487c-a742-3195fb220ff8/volumes"
Sep 19 09:03:55.262557 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:55.262519    2566 generic.go:356] "Generic (PLEG): container finished" podID="f02c240a-0127-48ea-983c-c569e96272c5" containerID="2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f" exitCode=0
Sep 19 09:03:55.262557 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:55.262556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerDied","Data":"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"}
Sep 19 09:03:56.268548 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:56.268513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerStarted","Data":"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"}
Sep 19 09:03:56.268953 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:56.268567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerStarted","Data":"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"}
Sep 19 09:03:56.268953 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:56.268752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:56.286530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:56.286486    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podStartSLOduration=2.597665665 podStartE2EDuration="6.286473251s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688807586s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:03:56.285349114 +0000 UTC m=+2600.606020702" watchObservedRunningTime="2026-09-19 09:03:56.286473251 +0000 UTC m=+2600.607144900"
Sep 19 09:03:57.271924 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:57.271889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:03:57.273161 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:57.273135    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:03:58.275082 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:03:58.275009    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:03.279789 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:03.279753    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:04:03.280240 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:03.280216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:13.280870 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:13.280826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:23.280724 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:23.280683    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:33.280467 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:33.280425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:43.280396 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:43.280356    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:04:53.280565 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:04:53.280524    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:05:03.280917 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:03.280880    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:05:10.662982 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:10.662950    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:05:10.663416 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:10.663198    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" containerID="cri-o://ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4" gracePeriod=30
Sep 19 09:05:10.663416 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:10.663283    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kube-rbac-proxy" containerID="cri-o://6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d" gracePeriod=30
Sep 19 09:05:10.685092 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:10.685063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:11.556906 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:11.556872    2566 generic.go:356] "Generic (PLEG): container finished" podID="f02c240a-0127-48ea-983c-c569e96272c5" containerID="6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d" exitCode=2
Sep 19 09:05:11.557072 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:11.556951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerDied","Data":"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"}
Sep 19 09:05:13.276258 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:13.276223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 19 09:05:13.280500 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:13.280479    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 09:05:14.499162 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.499139    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:05:14.519299 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519269    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert\") pod \"f02c240a-0127-48ea-983c-c569e96272c5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") "
Sep 19 09:05:14.519461 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls\") pod \"f02c240a-0127-48ea-983c-c569e96272c5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") "
Sep 19 09:05:14.519461 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519354    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xtrj\" (UniqueName: \"kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj\") pod \"f02c240a-0127-48ea-983c-c569e96272c5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") "
Sep 19 09:05:14.519461 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location\") pod \"f02c240a-0127-48ea-983c-c569e96272c5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") "
Sep 19 09:05:14.519630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config\") pod \"f02c240a-0127-48ea-983c-c569e96272c5\" (UID: \"f02c240a-0127-48ea-983c-c569e96272c5\") "
Sep 19 09:05:14.519684 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f02c240a-0127-48ea-983c-c569e96272c5" (UID: "f02c240a-0127-48ea-983c-c569e96272c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:14.519740 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f02c240a-0127-48ea-983c-c569e96272c5" (UID: "f02c240a-0127-48ea-983c-c569e96272c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:05:14.519915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519886    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:05:14.519915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519908    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02c240a-0127-48ea-983c-c569e96272c5-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:05:14.520056 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.519900    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f02c240a-0127-48ea-983c-c569e96272c5" (UID: "f02c240a-0127-48ea-983c-c569e96272c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:05:14.521684 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.521655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f02c240a-0127-48ea-983c-c569e96272c5" (UID: "f02c240a-0127-48ea-983c-c569e96272c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:05:14.521973 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.521945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj" (OuterVolumeSpecName: "kube-api-access-5xtrj") pod "f02c240a-0127-48ea-983c-c569e96272c5" (UID: "f02c240a-0127-48ea-983c-c569e96272c5"). InnerVolumeSpecName "kube-api-access-5xtrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:05:14.568822 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.568796    2566 generic.go:356] "Generic (PLEG): container finished" podID="f02c240a-0127-48ea-983c-c569e96272c5" containerID="ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4" exitCode=0
Sep 19 09:05:14.568915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.568831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerDied","Data":"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"}
Sep 19 09:05:14.568915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.568848    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"
Sep 19 09:05:14.568915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.568879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5" event={"ID":"f02c240a-0127-48ea-983c-c569e96272c5","Type":"ContainerDied","Data":"8b2dc046210653fa15c25005e6bddc68362b79b1ef49e8b6566005906482f9a9"}
Sep 19 09:05:14.568915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.568913    2566 scope.go:117] "RemoveContainer" containerID="6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"
Sep 19 09:05:14.579612 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.579596    2566 scope.go:117] "RemoveContainer" containerID="ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"
Sep 19 09:05:14.586564 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.586549    2566 scope.go:117] "RemoveContainer" containerID="2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"
Sep 19 09:05:14.591513 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.591492    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:05:14.593564 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.593544    2566 scope.go:117] "RemoveContainer" containerID="6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"
Sep 19 09:05:14.594156 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:14.594049    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d\": container with ID starting with 6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d not found: ID does not exist" containerID="6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"
Sep 19 09:05:14.594156 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.594083    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d"} err="failed to get container status \"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d\": rpc error: code = NotFound desc = could not find container \"6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d\": container with ID starting with 6ef0d025cdfedb5b1a096839bd460caedb68cec2f638ef7ef73f4157f823e78d not found: ID does not exist"
Sep 19 09:05:14.594324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.594165    2566 scope.go:117] "RemoveContainer" containerID="ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"
Sep 19 09:05:14.594465 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:14.594449    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4\": container with ID starting with ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4 not found: ID does not exist" containerID="ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"
Sep 19 09:05:14.594509 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.594469    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4"} err="failed to get container status \"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4\": rpc error: code = NotFound desc = could not find container \"ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4\": container with ID starting with ec1d9a8cccd7896ba65b73daa4639ab48d0a66a3427948850b1c2d7dd1a18bc4 not found: ID does not exist"
Sep 19 09:05:14.594509 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.594483    2566 scope.go:117] "RemoveContainer" containerID="2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"
Sep 19 09:05:14.594733 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:14.594708    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f\": container with ID starting with 2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f not found: ID does not exist" containerID="2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"
Sep 19 09:05:14.594802 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.594743    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f"} err="failed to get container status \"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f\": rpc error: code = NotFound desc = could not find container \"2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f\": container with ID starting with 2533a463bc9f426d86367c86d1afee019cca0b646019ad9e67ae93dcad10d47f not found: ID does not exist"
Sep 19 09:05:14.595380 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.595364    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-98lf5"]
Sep 19 09:05:14.621208 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.621191    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtrj\" (UniqueName: \"kubernetes.io/projected/f02c240a-0127-48ea-983c-c569e96272c5-kube-api-access-5xtrj\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:05:14.621293 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.621209    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02c240a-0127-48ea-983c-c569e96272c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:05:14.621293 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:14.621220    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02c240a-0127-48ea-983c-c569e96272c5-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:05:15.811282 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.811249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:15.844430 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844401    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:05:15.844659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kserve-container"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844661    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kserve-container"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844670    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kube-rbac-proxy"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844676    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kube-rbac-proxy"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844685    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="storage-initializer"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844691    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="storage-initializer"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844698    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kube-rbac-proxy"
Sep 19 09:05:15.844707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844704    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kube-rbac-proxy"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844711    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844719    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844730    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="storage-initializer"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844735    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="storage-initializer"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844791    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kube-rbac-proxy"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844801    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kserve-container"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844807    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02c240a-0127-48ea-983c-c569e96272c5" containerName="kserve-container"
Sep 19 09:05:15.844943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.844813    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5c733d-34a5-487c-a742-3195fb220ff8" containerName="kube-rbac-proxy"
Sep 19 09:05:15.849326 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.849308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:15.851920 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.851894    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:05:15.852016 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.851977    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:05:15.852916 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.852896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 19 09:05:15.852916 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.852905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:05:15.853077 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.852915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:05:15.853077 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.852899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:05:15.853077 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.852939    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:05:15.858992 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.858970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:05:15.931259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.931229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:15.931387 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.931300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:15.931387 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.931351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxn5p\" (UniqueName: \"kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:15.931506 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.931393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:15.931506 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:15.931446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxn5p\" (UniqueName: \"kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:16.032156    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032204 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:16.032204    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls podName:42f90390-677e-4135-a431-858de47e9010 nodeName:}" failed. No retries permitted until 2026-09-19 09:05:16.532189431 +0000 UTC m=+2680.852860996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" (UID: "42f90390-677e-4135-a431-858de47e9010") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 19 09:05:16.032614 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032832 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.032879 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.032849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.040945 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.040925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxn5p\" (UniqueName: \"kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.180737 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.180700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:16.181244 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.181223    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02c240a-0127-48ea-983c-c569e96272c5" path="/var/lib/kubelet/pods/f02c240a-0127-48ea-983c-c569e96272c5/volumes"
Sep 19 09:05:16.535531 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.535455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.537856 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.537839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:16.759975 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:16.759929    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:17.034825 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:17.034796    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:05:17.037251 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:05:17.037223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f90390_677e_4135_a431_858de47e9010.slice/crio-ec80943f32192fd4e99d062d919c54368246fbe17860f3a19717adb1214c52d3 WatchSource:0}: Error finding container ec80943f32192fd4e99d062d919c54368246fbe17860f3a19717adb1214c52d3: Status 404 returned error can't find the container with id ec80943f32192fd4e99d062d919c54368246fbe17860f3a19717adb1214c52d3
Sep 19 09:05:17.582586 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:17.582542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerStarted","Data":"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"}
Sep 19 09:05:17.582750 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:17.582620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerStarted","Data":"ec80943f32192fd4e99d062d919c54368246fbe17860f3a19717adb1214c52d3"}
Sep 19 09:05:21.599470 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:21.599439    2566 generic.go:356] "Generic (PLEG): container finished" podID="42f90390-677e-4135-a431-858de47e9010" containerID="72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec" exitCode=0
Sep 19 09:05:21.599866 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:21.599521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerDied","Data":"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"}
Sep 19 09:05:22.605493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:22.605461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerStarted","Data":"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"}
Sep 19 09:05:22.606011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:22.605531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerStarted","Data":"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"}
Sep 19 09:05:22.606011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:22.605792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:22.625395 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:22.625336    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podStartSLOduration=3.719103469 podStartE2EDuration="7.625320069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.9062166s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:05:22.624130435 +0000 UTC m=+2686.944802023" watchObservedRunningTime="2026-09-19 09:05:22.625320069 +0000 UTC m=+2686.945991657"
Sep 19 09:05:23.609396 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:23.609368    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:28.015382 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:28.015349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:29.618308 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:29.618283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:05:33.143858 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.143823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:33.173791 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.173754    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:05:33.177213 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.177199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.179948 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.179931    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 19 09:05:33.180038 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.179988    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:05:33.187502 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.187479    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:05:33.259947 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.259922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.260055 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.259959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.260055 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.259995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.260055 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.260013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.260055 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.260046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.360818 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.360795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.360940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.360834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.360940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.360868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.360940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.360900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.360940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.360924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.361122 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:33.360961    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 19 09:05:33.361122 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:05:33.361027    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls podName:bf80005e-5a46-4d4f-8068-acd7a92871b0 nodeName:}" failed. No retries permitted until 2026-09-19 09:05:33.861007508 +0000 UTC m=+2698.181679073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 19 09:05:33.361343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.361325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.361492 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.361476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.361532 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.361492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.372379 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.372360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.459001 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.458922    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:05:33.864852 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.864820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:33.867212 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:33.867194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:34.086569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:34.086540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:34.358577 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:34.358520    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:05:34.360786 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:05:34.360742    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf80005e_5a46_4d4f_8068_acd7a92871b0.slice/crio-ad4be0473ab6e02bb61c52e5b47a1ae36555aa163f6126ec32fd1070645065a6 WatchSource:0}: Error finding container ad4be0473ab6e02bb61c52e5b47a1ae36555aa163f6126ec32fd1070645065a6: Status 404 returned error can't find the container with id ad4be0473ab6e02bb61c52e5b47a1ae36555aa163f6126ec32fd1070645065a6
Sep 19 09:05:34.649725 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:34.649638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerStarted","Data":"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"}
Sep 19 09:05:34.649725 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:34.649692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerStarted","Data":"ad4be0473ab6e02bb61c52e5b47a1ae36555aa163f6126ec32fd1070645065a6"}
Sep 19 09:05:38.668965 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:38.668895    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerID="9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9" exitCode=0
Sep 19 09:05:38.669282 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:38.668970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerDied","Data":"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"}
Sep 19 09:05:39.676084 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:39.676047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerStarted","Data":"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"}
Sep 19 09:05:39.676454 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:39.676137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerStarted","Data":"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"}
Sep 19 09:05:39.676454 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:39.676340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:39.696738 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:39.696677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" podStartSLOduration=2.73685906 podStartE2EDuration="6.696657605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959798545s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:05:39.695233824 +0000 UTC m=+2704.015905414" watchObservedRunningTime="2026-09-19 09:05:39.696657605 +0000 UTC m=+2704.017329196"
Sep 19 09:05:40.680201 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:40.680174    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:46.689594 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:46.689565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:05:59.643173 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:05:59.643125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 19 09:06:09.621488 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:09.621459    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:06:14.483754 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:14.483707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:15.909157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:15.909121    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:06:15.909564 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:15.909377    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container" containerID="cri-o://dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a" gracePeriod=30
Sep 19 09:06:15.909564 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:15.909416    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kube-rbac-proxy" containerID="cri-o://94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f" gracePeriod=30
Sep 19 09:06:15.932990 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:15.932954    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:16.693459 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:16.693426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:06:16.815637 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:16.815607    2566 generic.go:356] "Generic (PLEG): container finished" podID="42f90390-677e-4135-a431-858de47e9010" containerID="94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f" exitCode=2
Sep 19 09:06:16.815811 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:16.815678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerDied","Data":"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"}
Sep 19 09:06:19.613368 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:19.613328    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 19 09:06:19.619434 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:19.619408    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 09:06:21.071567 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:21.071537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:21.222534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:21.222498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:22.137970 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.137951    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:06:22.315218 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location\") pod \"42f90390-677e-4135-a431-858de47e9010\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") "
Sep 19 09:06:22.315218 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert\") pod \"42f90390-677e-4135-a431-858de47e9010\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") "
Sep 19 09:06:22.315218 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config\") pod \"42f90390-677e-4135-a431-858de47e9010\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") "
Sep 19 09:06:22.315485 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315233    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxn5p\" (UniqueName: \"kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p\") pod \"42f90390-677e-4135-a431-858de47e9010\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") "
Sep 19 09:06:22.315485 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") pod \"42f90390-677e-4135-a431-858de47e9010\" (UID: \"42f90390-677e-4135-a431-858de47e9010\") "
Sep 19 09:06:22.315485 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42f90390-677e-4135-a431-858de47e9010" (UID: "42f90390-677e-4135-a431-858de47e9010"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:22.315641 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315594    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42f90390-677e-4135-a431-858de47e9010" (UID: "42f90390-677e-4135-a431-858de47e9010"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:22.315641 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315626    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42f90390-677e-4135-a431-858de47e9010" (UID: "42f90390-677e-4135-a431-858de47e9010"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:22.315722 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.315637    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42f90390-677e-4135-a431-858de47e9010-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:22.317304 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.317281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42f90390-677e-4135-a431-858de47e9010" (UID: "42f90390-677e-4135-a431-858de47e9010"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:06:22.317376 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.317360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p" (OuterVolumeSpecName: "kube-api-access-vxn5p") pod "42f90390-677e-4135-a431-858de47e9010" (UID: "42f90390-677e-4135-a431-858de47e9010"). InnerVolumeSpecName "kube-api-access-vxn5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:06:22.416696 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.416672    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:22.416696 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.416698    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f90390-677e-4135-a431-858de47e9010-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:22.416880 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.416711    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxn5p\" (UniqueName: \"kubernetes.io/projected/42f90390-677e-4135-a431-858de47e9010-kube-api-access-vxn5p\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:22.416880 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.416723    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f90390-677e-4135-a431-858de47e9010-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:22.840322 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.840286    2566 generic.go:356] "Generic (PLEG): container finished" podID="42f90390-677e-4135-a431-858de47e9010" containerID="dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a" exitCode=0
Sep 19 09:06:22.840518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.840334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerDied","Data":"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"}
Sep 19 09:06:22.840518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.840350    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"
Sep 19 09:06:22.840518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.840382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b" event={"ID":"42f90390-677e-4135-a431-858de47e9010","Type":"ContainerDied","Data":"ec80943f32192fd4e99d062d919c54368246fbe17860f3a19717adb1214c52d3"}
Sep 19 09:06:22.840518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.840409    2566 scope.go:117] "RemoveContainer" containerID="94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"
Sep 19 09:06:22.849053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.849036    2566 scope.go:117] "RemoveContainer" containerID="dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"
Sep 19 09:06:22.855948 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.855927    2566 scope.go:117] "RemoveContainer" containerID="72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"
Sep 19 09:06:22.861907 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.861886    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:06:22.862637 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.862623    2566 scope.go:117] "RemoveContainer" containerID="94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"
Sep 19 09:06:22.862895 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:22.862877    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f\": container with ID starting with 94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f not found: ID does not exist" containerID="94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"
Sep 19 09:06:22.862952 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.862903    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f"} err="failed to get container status \"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f\": rpc error: code = NotFound desc = could not find container \"94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f\": container with ID starting with 94c2784eed96888e230100d4eaf644b1cd9bec145519337cace85c84779c566f not found: ID does not exist"
Sep 19 09:06:22.862952 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.862919    2566 scope.go:117] "RemoveContainer" containerID="dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"
Sep 19 09:06:22.863155 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:22.863138    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a\": container with ID starting with dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a not found: ID does not exist" containerID="dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"
Sep 19 09:06:22.863194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.863160    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a"} err="failed to get container status \"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a\": rpc error: code = NotFound desc = could not find container \"dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a\": container with ID starting with dc78f0eed542d8615fb6c02094ce1e248ff17f03bc9b6592b4e378abc78ae32a not found: ID does not exist"
Sep 19 09:06:22.863194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.863173    2566 scope.go:117] "RemoveContainer" containerID="72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"
Sep 19 09:06:22.863396 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:22.863381    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec\": container with ID starting with 72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec not found: ID does not exist" containerID="72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"
Sep 19 09:06:22.863445 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.863398    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec"} err="failed to get container status \"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec\": rpc error: code = NotFound desc = could not find container \"72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec\": container with ID starting with 72a6aab85afd5910d0df79675bdf92efd7ed7044de7b8753adc8f49af813d9ec not found: ID does not exist"
Sep 19 09:06:22.867741 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:22.867722    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cf46b"]
Sep 19 09:06:23.246009 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.245978    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:06:23.246386 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.246226    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kserve-container" containerID="cri-o://e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b" gracePeriod=30
Sep 19 09:06:23.246386 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.246256    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kube-rbac-proxy" containerID="cri-o://425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049" gracePeriod=30
Sep 19 09:06:23.263076 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.263041    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:23.845918 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.845885    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerID="425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049" exitCode=2
Sep 19 09:06:23.846099 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:23.845960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerDied","Data":"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"}
Sep 19 09:06:24.180271 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:24.180241    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f90390-677e-4135-a431-858de47e9010" path="/var/lib/kubelet/pods/42f90390-677e-4135-a431-858de47e9010/volumes"
Sep 19 09:06:26.685271 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:26.685231    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 19 09:06:27.496012 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:27.495954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/isvc-lightgbm-v2-runtime/ready\": read tcp 10.133.0.2:58202->10.133.0.46:8080: read: connection reset by peer"
Sep 19 09:06:28.083510 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.083487    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:06:28.160205 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160133    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r\") pod \"bf80005e-5a46-4d4f-8068-acd7a92871b0\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") "
Sep 19 09:06:28.160349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") pod \"bf80005e-5a46-4d4f-8068-acd7a92871b0\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") "
Sep 19 09:06:28.160349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config\") pod \"bf80005e-5a46-4d4f-8068-acd7a92871b0\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") "
Sep 19 09:06:28.160349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location\") pod \"bf80005e-5a46-4d4f-8068-acd7a92871b0\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") "
Sep 19 09:06:28.160349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert\") pod \"bf80005e-5a46-4d4f-8068-acd7a92871b0\" (UID: \"bf80005e-5a46-4d4f-8068-acd7a92871b0\") "
Sep 19 09:06:28.160650 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160615    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf80005e-5a46-4d4f-8068-acd7a92871b0" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:28.160650 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160612    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf80005e-5a46-4d4f-8068-acd7a92871b0" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:28.160784 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.160656    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf80005e-5a46-4d4f-8068-acd7a92871b0" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:28.162183 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.162163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r" (OuterVolumeSpecName: "kube-api-access-f9d8r") pod "bf80005e-5a46-4d4f-8068-acd7a92871b0" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0"). InnerVolumeSpecName "kube-api-access-f9d8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:06:28.162246 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.162188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf80005e-5a46-4d4f-8068-acd7a92871b0" (UID: "bf80005e-5a46-4d4f-8068-acd7a92871b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:06:28.261063 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.261035    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-api-access-f9d8r\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:28.261063 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.261060    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf80005e-5a46-4d4f-8068-acd7a92871b0-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:28.261063 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.261070    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:28.261250 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.261079    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf80005e-5a46-4d4f-8068-acd7a92871b0-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:28.261250 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.261087    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf80005e-5a46-4d4f-8068-acd7a92871b0-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:28.373894 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.373854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:28.401001 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.400975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:06:28.401228 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401216    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="storage-initializer"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401229    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="storage-initializer"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401236    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="storage-initializer"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401241    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="storage-initializer"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401247    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kserve-container"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401252    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kserve-container"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kube-rbac-proxy"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kube-rbac-proxy"
Sep 19 09:06:28.401272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401272    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kube-rbac-proxy"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401278    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kube-rbac-proxy"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401288    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401293    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kserve-container"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401344    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerName="kube-rbac-proxy"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kserve-container"
Sep 19 09:06:28.401538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.401356    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f90390-677e-4135-a431-858de47e9010" containerName="kube-rbac-proxy"
Sep 19 09:06:28.404303 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.404289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.406664 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.406647    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 19 09:06:28.406943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.406923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 19 09:06:28.414020 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.413974    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:06:28.463210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.463179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.463346 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.463234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.463346 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.463287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.463439 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.463347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.463439 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.463364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.536394 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.536365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:28.564203 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564291 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564291 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564291 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564434 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564624 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564881 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.564940 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.564919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.566648 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.566628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.571523 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.571505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.714028 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.713949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:28.868331 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.868300    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf80005e-5a46-4d4f-8068-acd7a92871b0" containerID="e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b" exitCode=0
Sep 19 09:06:28.868483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.868369    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"
Sep 19 09:06:28.868483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.868371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerDied","Data":"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"}
Sep 19 09:06:28.868595 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.868495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz" event={"ID":"bf80005e-5a46-4d4f-8068-acd7a92871b0","Type":"ContainerDied","Data":"ad4be0473ab6e02bb61c52e5b47a1ae36555aa163f6126ec32fd1070645065a6"}
Sep 19 09:06:28.868595 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.868525    2566 scope.go:117] "RemoveContainer" containerID="425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"
Sep 19 09:06:28.875852 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.875831    2566 scope.go:117] "RemoveContainer" containerID="e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"
Sep 19 09:06:28.882564 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.882548    2566 scope.go:117] "RemoveContainer" containerID="9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"
Sep 19 09:06:28.887232 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.887210    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:06:28.890932 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.890916    2566 scope.go:117] "RemoveContainer" containerID="425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"
Sep 19 09:06:28.891181 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891163    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-thvnz"]
Sep 19 09:06:28.891231 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:28.891197    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049\": container with ID starting with 425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049 not found: ID does not exist" containerID="425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"
Sep 19 09:06:28.891275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049"} err="failed to get container status \"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049\": rpc error: code = NotFound desc = could not find container \"425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049\": container with ID starting with 425740e76a23eaaa04114662da019c942e409fa0637ef1c8a49bd55fd8d2f049 not found: ID does not exist"
Sep 19 09:06:28.891275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891239    2566 scope.go:117] "RemoveContainer" containerID="e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"
Sep 19 09:06:28.891508 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:28.891495    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b\": container with ID starting with e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b not found: ID does not exist" containerID="e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"
Sep 19 09:06:28.891550 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891512    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b"} err="failed to get container status \"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b\": rpc error: code = NotFound desc = could not find container \"e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b\": container with ID starting with e39a1cedb3a0ba0ef016465965182dd9d025f6b09214b2ef7822d5c9a965a24b not found: ID does not exist"
Sep 19 09:06:28.891550 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891525    2566 scope.go:117] "RemoveContainer" containerID="9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"
Sep 19 09:06:28.891736 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:06:28.891720    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9\": container with ID starting with 9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9 not found: ID does not exist" containerID="9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"
Sep 19 09:06:28.891793 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.891738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9"} err="failed to get container status \"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9\": rpc error: code = NotFound desc = could not find container \"9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9\": container with ID starting with 9b312a11989e5aaa8d7c117bada00ba87331452acc4edf0a76f51a1406a463f9 not found: ID does not exist"
Sep 19 09:06:28.982585 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:28.982562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:06:28.984834 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:06:28.984813    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29852ef8_e51d_4edf_b015_f7a7e56cb6c7.slice/crio-25033a6de23facf1a4bd77b275e7e77d76f45bce61544fe27d21119ad3229f59 WatchSource:0}: Error finding container 25033a6de23facf1a4bd77b275e7e77d76f45bce61544fe27d21119ad3229f59: Status 404 returned error can't find the container with id 25033a6de23facf1a4bd77b275e7e77d76f45bce61544fe27d21119ad3229f59
Sep 19 09:06:29.876274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:29.876238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerStarted","Data":"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"}
Sep 19 09:06:29.876722 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:29.876306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerStarted","Data":"25033a6de23facf1a4bd77b275e7e77d76f45bce61544fe27d21119ad3229f59"}
Sep 19 09:06:30.180072 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:30.180001    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf80005e-5a46-4d4f-8068-acd7a92871b0" path="/var/lib/kubelet/pods/bf80005e-5a46-4d4f-8068-acd7a92871b0/volumes"
Sep 19 09:06:32.889276 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:32.889243    2566 generic.go:356] "Generic (PLEG): container finished" podID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerID="d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7" exitCode=0
Sep 19 09:06:32.889665 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:32.889317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerDied","Data":"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"}
Sep 19 09:06:33.895646 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.895609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerStarted","Data":"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"}
Sep 19 09:06:33.896065 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.895673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerStarted","Data":"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"}
Sep 19 09:06:33.896065 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.895962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:33.896148 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.896093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:33.897572 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.897546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 09:06:33.915308 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:33.915256    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podStartSLOduration=2.266784091 podStartE2EDuration="5.915244414s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648460323s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:33.913259862 +0000 UTC m=+2758.233931449" watchObservedRunningTime="2026-09-19 09:06:33.915244414 +0000 UTC m=+2758.235916000"
Sep 19 09:06:34.899657 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:34.899614    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 09:06:39.905018 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:39.904984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:39.905609 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:39.905579    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 09:06:49.905965 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:49.905930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:58.511194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.511109    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:06:58.511639 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.511339    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container" containerID="cri-o://fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0" gracePeriod=30
Sep 19 09:06:58.511639 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.511424    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kube-rbac-proxy" containerID="cri-o://1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7" gracePeriod=30
Sep 19 09:06:58.535741 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.535716    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:06:58.996024 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.995994    2566 generic.go:356] "Generic (PLEG): container finished" podID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerID="1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7" exitCode=2
Sep 19 09:06:58.996183 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:58.996031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerDied","Data":"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"}
Sep 19 09:06:59.153862 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.153839    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:06:59.288932 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.288844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config\") pod \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") "
Sep 19 09:06:59.288932 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.288924    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location\") pod \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") "
Sep 19 09:06:59.289132 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.288977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls\") pod \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") "
Sep 19 09:06:59.289132 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.289004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth\") pod \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") "
Sep 19 09:06:59.289132 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.289033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert\") pod \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\" (UID: \"29852ef8-e51d-4edf-b015-f7a7e56cb6c7\") "
Sep 19 09:06:59.289279 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.289248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29852ef8-e51d-4edf-b015-f7a7e56cb6c7" (UID: "29852ef8-e51d-4edf-b015-f7a7e56cb6c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:59.289406 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.289317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29852ef8-e51d-4edf-b015-f7a7e56cb6c7" (UID: "29852ef8-e51d-4edf-b015-f7a7e56cb6c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:59.289464 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.289443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29852ef8-e51d-4edf-b015-f7a7e56cb6c7" (UID: "29852ef8-e51d-4edf-b015-f7a7e56cb6c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:59.291147 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.291124    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth" (OuterVolumeSpecName: "kube-api-access-l9qth") pod "29852ef8-e51d-4edf-b015-f7a7e56cb6c7" (UID: "29852ef8-e51d-4edf-b015-f7a7e56cb6c7"). InnerVolumeSpecName "kube-api-access-l9qth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:06:59.291210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.291192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29852ef8-e51d-4edf-b015-f7a7e56cb6c7" (UID: "29852ef8-e51d-4edf-b015-f7a7e56cb6c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:06:59.390409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.390376    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:59.390409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.390405    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-api-access-l9qth\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:59.390409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.390416    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:59.390637 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.390425    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:06:59.390637 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:06:59.390433    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29852ef8-e51d-4edf-b015-f7a7e56cb6c7-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:00.002376 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.002344    2566 generic.go:356] "Generic (PLEG): container finished" podID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerID="fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0" exitCode=0
Sep 19 09:07:00.002807 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.002407    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"
Sep 19 09:07:00.002807 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.002436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerDied","Data":"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"}
Sep 19 09:07:00.002807 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.002506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt" event={"ID":"29852ef8-e51d-4edf-b015-f7a7e56cb6c7","Type":"ContainerDied","Data":"25033a6de23facf1a4bd77b275e7e77d76f45bce61544fe27d21119ad3229f59"}
Sep 19 09:07:00.002807 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.002543    2566 scope.go:117] "RemoveContainer" containerID="1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"
Sep 19 09:07:00.011693 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.011674    2566 scope.go:117] "RemoveContainer" containerID="fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"
Sep 19 09:07:00.018963 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.018944    2566 scope.go:117] "RemoveContainer" containerID="d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"
Sep 19 09:07:00.024843 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.024820    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:07:00.026072 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.026057    2566 scope.go:117] "RemoveContainer" containerID="1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"
Sep 19 09:07:00.026299 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:00.026282    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7\": container with ID starting with 1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7 not found: ID does not exist" containerID="1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"
Sep 19 09:07:00.026373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.026306    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7"} err="failed to get container status \"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7\": rpc error: code = NotFound desc = could not find container \"1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7\": container with ID starting with 1c078bb314bc7123e7d097b8bf8be0b15224f1cf97dea79b2152aeb49763b4c7 not found: ID does not exist"
Sep 19 09:07:00.026373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.026321    2566 scope.go:117] "RemoveContainer" containerID="fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"
Sep 19 09:07:00.026698 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:00.026654    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0\": container with ID starting with fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0 not found: ID does not exist" containerID="fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"
Sep 19 09:07:00.026698 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.026685    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0"} err="failed to get container status \"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0\": rpc error: code = NotFound desc = could not find container \"fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0\": container with ID starting with fcc4a4a24dd690d4f064bbe474ac7038bfd6e2f1862c2880e0bc060cc072f1c0 not found: ID does not exist"
Sep 19 09:07:00.026938 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.026705    2566 scope.go:117] "RemoveContainer" containerID="d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"
Sep 19 09:07:00.027110 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:00.027015    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7\": container with ID starting with d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7 not found: ID does not exist" containerID="d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"
Sep 19 09:07:00.027110 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.027043    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7"} err="failed to get container status \"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7\": rpc error: code = NotFound desc = could not find container \"d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7\": container with ID starting with d13c364da81bb3c741227572320706914c0bbfda94cf1b1ee72e7d2181b3a9a7 not found: ID does not exist"
Sep 19 09:07:00.028307 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.028288    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4twbt"]
Sep 19 09:07:00.180263 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:00.180234    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" path="/var/lib/kubelet/pods/29852ef8-e51d-4edf-b015-f7a7e56cb6c7/volumes"
Sep 19 09:07:03.685696 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.685661    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:07:03.712897 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.712866    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:03.713126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713114    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kube-rbac-proxy"
Sep 19 09:07:03.713126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713127    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kube-rbac-proxy"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713141    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="storage-initializer"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713147    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="storage-initializer"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713158    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713164    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kserve-container"
Sep 19 09:07:03.713210 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.713211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29852ef8-e51d-4edf-b015-f7a7e56cb6c7" containerName="kube-rbac-proxy"
Sep 19 09:07:03.717445 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.717426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.719764 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.719740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:07:03.719887 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.719873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:07:03.719946 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.719922    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:07:03.720053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.720035    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:07:03.720524 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.720509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:07:03.720619 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.720582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 19 09:07:03.720674 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.720585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:07:03.727470 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.727450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:03.821628 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.821589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.821834 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.821638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrhj\" (UniqueName: \"kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.821834 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.821658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.821834 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.821736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.821834 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.821796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.922990 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.922954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923152 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrhj\" (UniqueName: \"kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923152 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923152 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923152 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923332 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:03.923196    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 19 09:07:03.923332 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:03.923270    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls podName:0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2 nodeName:}" failed. No retries permitted until 2026-09-19 09:07:04.423248677 +0000 UTC m=+2788.743920257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 19 09:07:03.923458 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923747 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.923797 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.923752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:03.933318 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:03.933286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrhj\" (UniqueName: \"kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:04.128071 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:04.128029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:07:04.428180 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:04.428104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:04.430405 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:04.430385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:04.628425 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:04.628376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:04.900748 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:04.900724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:04.903238 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:07:04.903210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3ddea7_6f81_4ff0_bdaf_f90f3e0b6cc2.slice/crio-872518f07e55909024b2852bf1aaab8604ae7301abb525144b4b727017a0eba9 WatchSource:0}: Error finding container 872518f07e55909024b2852bf1aaab8604ae7301abb525144b4b727017a0eba9: Status 404 returned error can't find the container with id 872518f07e55909024b2852bf1aaab8604ae7301abb525144b4b727017a0eba9
Sep 19 09:07:05.024634 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:05.024603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerStarted","Data":"7c30d4254ddd77d3210a2af52499d9c71cbfe2f750d439464ac9d220a37832fa"}
Sep 19 09:07:05.024827 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:05.024666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerStarted","Data":"872518f07e55909024b2852bf1aaab8604ae7301abb525144b4b727017a0eba9"}
Sep 19 09:07:10.044658 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:10.044624    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerID="7c30d4254ddd77d3210a2af52499d9c71cbfe2f750d439464ac9d220a37832fa" exitCode=0
Sep 19 09:07:10.044658 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:10.044664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerDied","Data":"7c30d4254ddd77d3210a2af52499d9c71cbfe2f750d439464ac9d220a37832fa"}
Sep 19 09:07:11.050809 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:11.050758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerStarted","Data":"70081b9e31cb8b6dfda025de07af19dc279465517e5724d8ed30badb1eb50356"}
Sep 19 09:07:11.051257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:11.050835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerStarted","Data":"cdf4061d60022c7d7e88b2865794dac5da4ed9d7490f3c47405a3229a426d4a5"}
Sep 19 09:07:11.051257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:11.051042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:11.051257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:11.051090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:11.071188 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:11.071142    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" podStartSLOduration=3.915682455 podStartE2EDuration="8.071127733s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.155445278s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:07:11.069160603 +0000 UTC m=+2795.389832191" watchObservedRunningTime="2026-09-19 09:07:11.071127733 +0000 UTC m=+2795.391799321"
Sep 19 09:07:17.060737 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:17.060706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:47.064933 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:47.064898    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:53.779341 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:53.779292    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:53.779860 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:53.779646    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kserve-container" containerID="cri-o://cdf4061d60022c7d7e88b2865794dac5da4ed9d7490f3c47405a3229a426d4a5" gracePeriod=30
Sep 19 09:07:53.779860 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:53.779796    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kube-rbac-proxy" containerID="cri-o://70081b9e31cb8b6dfda025de07af19dc279465517e5724d8ed30badb1eb50356" gracePeriod=30
Sep 19 09:07:53.799801 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:53.799741    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:07:54.224651 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:54.224614    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerID="70081b9e31cb8b6dfda025de07af19dc279465517e5724d8ed30badb1eb50356" exitCode=2
Sep 19 09:07:54.224873 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:54.224683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerDied","Data":"70081b9e31cb8b6dfda025de07af19dc279465517e5724d8ed30badb1eb50356"}
Sep 19 09:07:55.232963 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.232925    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerID="cdf4061d60022c7d7e88b2865794dac5da4ed9d7490f3c47405a3229a426d4a5" exitCode=0
Sep 19 09:07:55.233372 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.232996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerDied","Data":"cdf4061d60022c7d7e88b2865794dac5da4ed9d7490f3c47405a3229a426d4a5"}
Sep 19 09:07:55.325168 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.325137    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:55.427617 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.427519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkrhj\" (UniqueName: \"kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj\") pod \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") "
Sep 19 09:07:55.427617 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.427564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") pod \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") "
Sep 19 09:07:55.427617 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.427591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location\") pod \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") "
Sep 19 09:07:55.427875 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.427631    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config\") pod \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") "
Sep 19 09:07:55.427875 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.427654    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert\") pod \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\" (UID: \"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2\") "
Sep 19 09:07:55.428081 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.428045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:07:55.428081 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.428070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:07:55.428244 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.428081    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:07:55.429726 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.429701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj" (OuterVolumeSpecName: "kube-api-access-lkrhj") pod "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2"). InnerVolumeSpecName "kube-api-access-lkrhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:07:55.429842 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.429724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" (UID: "0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:07:55.528234 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.528190    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrhj\" (UniqueName: \"kubernetes.io/projected/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-api-access-lkrhj\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:55.528234 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.528224    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:55.528234 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.528233    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:55.528234 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.528242    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:55.528512 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:55.528255    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:07:56.239556 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.239516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m" event={"ID":"0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2","Type":"ContainerDied","Data":"872518f07e55909024b2852bf1aaab8604ae7301abb525144b4b727017a0eba9"}
Sep 19 09:07:56.239556 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.239556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"
Sep 19 09:07:56.240094 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.239598    2566 scope.go:117] "RemoveContainer" containerID="70081b9e31cb8b6dfda025de07af19dc279465517e5724d8ed30badb1eb50356"
Sep 19 09:07:56.248042 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.248013    2566 scope.go:117] "RemoveContainer" containerID="cdf4061d60022c7d7e88b2865794dac5da4ed9d7490f3c47405a3229a426d4a5"
Sep 19 09:07:56.257221 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.257192    2566 scope.go:117] "RemoveContainer" containerID="7c30d4254ddd77d3210a2af52499d9c71cbfe2f750d439464ac9d220a37832fa"
Sep 19 09:07:56.257429 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.257409    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:56.261086 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:56.261055    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-dfh5m"]
Sep 19 09:07:58.181303 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:58.181262    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" path="/var/lib/kubelet/pods/0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2/volumes"
Sep 19 09:07:58.988064 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:58.987996    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:07:59.014011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.013953    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:07:59.014324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014307    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="storage-initializer"
Sep 19 09:07:59.014371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014327    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="storage-initializer"
Sep 19 09:07:59.014371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014350    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kserve-container"
Sep 19 09:07:59.014371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kserve-container"
Sep 19 09:07:59.014371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014364    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kube-rbac-proxy"
Sep 19 09:07:59.014371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014371    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kube-rbac-proxy"
Sep 19 09:07:59.014534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014433    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kube-rbac-proxy"
Sep 19 09:07:59.014534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.014444    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3ddea7-6f81-4ff0-bdaf-f90f3e0b6cc2" containerName="kserve-container"
Sep 19 09:07:59.020057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.020028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.022984 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.022955    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:07:59.023164 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.022961    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 19 09:07:59.023164 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.023016    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:07:59.023164 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.023160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:07:59.023392 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.023326    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:07:59.023392 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.023377    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 19 09:07:59.023496 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.023434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:07:59.028407 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.028290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:07:59.060990 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.060944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.061169 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.061003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.061169 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.061119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rt2\" (UniqueName: \"kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.061257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.061174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.061257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.061208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162354 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rt2\" (UniqueName: \"kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162354 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162615 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162615 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162615 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162888 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:59.162726    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 19 09:07:59.162888 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.162744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.162888 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:07:59.162819    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls podName:c13aab63-3e3a-4455-923d-2c8e43babc3b nodeName:}" failed. No retries permitted until 2026-09-19 09:07:59.66279761 +0000 UTC m=+2843.983469181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls") pod "isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 19 09:07:59.163265 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.163242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.163340 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.163250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.171862 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.171828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rt2\" (UniqueName: \"kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.449117 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.449078    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:07:59.667746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.667690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.670675 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.670647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:07:59.932592 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:07:59.932534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:08:00.216161 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:00.216128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:08:00.219058 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:08:00.219019    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13aab63_3e3a_4455_923d_2c8e43babc3b.slice/crio-84d462aa1c754cb3cb35899b00906ed08903485e0047d6bbec22c47b37124d89 WatchSource:0}: Error finding container 84d462aa1c754cb3cb35899b00906ed08903485e0047d6bbec22c47b37124d89: Status 404 returned error can't find the container with id 84d462aa1c754cb3cb35899b00906ed08903485e0047d6bbec22c47b37124d89
Sep 19 09:08:00.258348 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:00.258317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerStarted","Data":"84d462aa1c754cb3cb35899b00906ed08903485e0047d6bbec22c47b37124d89"}
Sep 19 09:08:01.264133 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:01.264082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerStarted","Data":"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"}
Sep 19 09:08:01.819525 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:01.819485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:08:05.282202 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:05.282167    2566 generic.go:356] "Generic (PLEG): container finished" podID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerID="7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc" exitCode=0
Sep 19 09:08:05.282581 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:05.282229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerDied","Data":"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"}
Sep 19 09:08:05.345518 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:05.345498    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:08:06.289933 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:06.289860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerStarted","Data":"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"}
Sep 19 09:08:07.087519 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:07.087481    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:08:07.330326 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:07.330292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:08:08.301373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.301337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerStarted","Data":"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"}
Sep 19 09:08:08.301533 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.301424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerStarted","Data":"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"}
Sep 19 09:08:08.301533 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.301454    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:08:08.301533 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.301479    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:08:08.301533 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.301502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:08:08.327363 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:08.327307    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podStartSLOduration=3.517282728 podStartE2EDuration="10.327286208s" totalImagesPullingTime="2.799251214s" totalInitContainerRuntime="4.010752266s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:08:08.324301684 +0000 UTC m=+2852.644973273" watchObservedRunningTime="2026-09-19 09:08:08.327286208 +0000 UTC m=+2852.647957796"
Sep 19 09:08:14.311324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:14.311294    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:08:33.444754 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:33.444657    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:08:39.933650 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:39.933605    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 19 09:08:44.456651 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:08:44.456619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:14.457450 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:14.457417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:19.075806 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.075756    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:09:19.076263 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.076043    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" containerID="cri-o://5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827" gracePeriod=30
Sep 19 09:09:19.076263 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.076066    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-agent" containerID="cri-o://a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff" gracePeriod=30
Sep 19 09:09:19.076263 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.076020    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container" containerID="cri-o://2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395" gracePeriod=30
Sep 19 09:09:19.104872 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.104835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:19.307153 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.307110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:19.587763 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.587732    2566 generic.go:356] "Generic (PLEG): container finished" podID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerID="5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827" exitCode=2
Sep 19 09:09:19.587952 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:19.587808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerDied","Data":"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"}
Sep 19 09:09:21.597958 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:21.597927    2566 generic.go:356] "Generic (PLEG): container finished" podID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerID="2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395" exitCode=0
Sep 19 09:09:21.598354 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:21.597998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerDied","Data":"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"}
Sep 19 09:09:24.294345 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.294303    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:24.306806 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.306745    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:24.312671 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.312637    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 19 09:09:24.372572 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.372536    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:09:24.375740 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.375716    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.379707 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.379667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 19 09:09:24.379913 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.379893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 19 09:09:24.394851 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.394817    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:09:24.434751 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.434662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.434751 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.434757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.435330 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.434892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59qfl\" (UniqueName: \"kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.435330 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.434959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.435330 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.435140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.489468 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.489433    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:24.536435 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.536630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.536630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.536630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59qfl\" (UniqueName: \"kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.536630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.536937 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.536913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.537222 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.537191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.537222 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.537202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.539135 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.539110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.547739 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.547675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59qfl\" (UniqueName: \"kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-8cqvz\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.685859 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.685815    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:24.965762 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:24.965728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:09:24.969170 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:09:24.969140    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8611586d_27b9_4175_bbd5_2bf6e0adf718.slice/crio-5b2088d46fe830b86b5a273a0992c7858b3e82fe1b523e94b3cabc81b72b7835 WatchSource:0}: Error finding container 5b2088d46fe830b86b5a273a0992c7858b3e82fe1b523e94b3cabc81b72b7835: Status 404 returned error can't find the container with id 5b2088d46fe830b86b5a273a0992c7858b3e82fe1b523e94b3cabc81b72b7835
Sep 19 09:09:25.614534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:25.614497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerStarted","Data":"579a95bcddc67e7425fb1d183a7ab18c405be382c38b77a9491aadad2f38f5a7"}
Sep 19 09:09:25.614931 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:25.614558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerStarted","Data":"5b2088d46fe830b86b5a273a0992c7858b3e82fe1b523e94b3cabc81b72b7835"}
Sep 19 09:09:29.306982 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:29.306931    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:29.307487 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:29.307059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:30.634696 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:30.634661    2566 generic.go:356] "Generic (PLEG): container finished" podID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerID="579a95bcddc67e7425fb1d183a7ab18c405be382c38b77a9491aadad2f38f5a7" exitCode=0
Sep 19 09:09:30.635123 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:30.634709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerDied","Data":"579a95bcddc67e7425fb1d183a7ab18c405be382c38b77a9491aadad2f38f5a7"}
Sep 19 09:09:34.307155 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:34.307105    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:34.312642 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:34.312605    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 19 09:09:37.661833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.661761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:37.802915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.802881    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:37.820057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.820021    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:09:37.853984 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.853948    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:09:37.854172 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.854092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:37.857461 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.857434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:09:37.857645 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.857600    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 19 09:09:37.857645 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.857609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 19 09:09:37.954345 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.954248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnvf\" (UniqueName: \"kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:37.954345 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.954307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:37.954589 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.954459    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:37.954589 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.954514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:37.954589 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:37.954564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055130 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055327 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055327 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055327 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055327 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnvf\" (UniqueName: \"kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055740 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.055969 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.056051 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.055999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.058037 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.058016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.061142 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.061117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:09:38.064351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.064326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnvf\" (UniqueName: \"kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:38.166036 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:38.165996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:39.307299 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:39.307248    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:41.954446 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:41.954410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:09:41.955450 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:09:41.955422    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf168c2a_443a_42bc_9207_e8a75e966b79.slice/crio-f0b999c2a93eb48892edafbc4bdcc0e5b40a468f4ee8c21bec3ddb30b936d93c WatchSource:0}: Error finding container f0b999c2a93eb48892edafbc4bdcc0e5b40a468f4ee8c21bec3ddb30b936d93c: Status 404 returned error can't find the container with id f0b999c2a93eb48892edafbc4bdcc0e5b40a468f4ee8c21bec3ddb30b936d93c
Sep 19 09:09:42.691208 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.691165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerStarted","Data":"e0702cb86ce045af1a12b65a25bbadfc52da8dfd7dc2e24f160e20ccb06e83cd"}
Sep 19 09:09:42.691427 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.691272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerStarted","Data":"3dd42b2bd8137480102ab47ab787f31a7bbcbfcc338e5b8f8dec1903872925eb"}
Sep 19 09:09:42.691526 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.691501    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:42.692956 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.692924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerStarted","Data":"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"}
Sep 19 09:09:42.693122 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.692976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerStarted","Data":"f0b999c2a93eb48892edafbc4bdcc0e5b40a468f4ee8c21bec3ddb30b936d93c"}
Sep 19 09:09:42.732739 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:42.732685    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podStartSLOduration=3.015189403 podStartE2EDuration="18.732664697s" totalImagesPullingTime="11.113347955s" totalInitContainerRuntime="4.604127339s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:09:42.731504891 +0000 UTC m=+2947.052176479" watchObservedRunningTime="2026-09-19 09:09:42.732664697 +0000 UTC m=+2947.053336342"
Sep 19 09:09:43.698946 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:43.698904    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerID="b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e" exitCode=0
Sep 19 09:09:43.699353 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:43.698988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerDied","Data":"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"}
Sep 19 09:09:43.699419 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:43.699395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:43.700908 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:43.700881    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:09:44.306492 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.306441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 19 09:09:44.311939 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.311908    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 19 09:09:44.312034 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.312023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:44.705416 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.705380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerStarted","Data":"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"}
Sep 19 09:09:44.705962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.705446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerStarted","Data":"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"}
Sep 19 09:09:44.705962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.705945    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:09:44.730830 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:44.730759    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podStartSLOduration=6.245467297 podStartE2EDuration="7.730742582s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.485275285s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:09:44.728668971 +0000 UTC m=+2949.049340560" watchObservedRunningTime="2026-09-19 09:09:44.730742582 +0000 UTC m=+2949.051414172"
Sep 19 09:09:45.709480 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:45.709442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:45.709894 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:45.709547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:45.711011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:45.710979    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:09:46.712725 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:46.712684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:09:49.225548 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.225512    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:49.354402 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") pod \"c13aab63-3e3a-4455-923d-2c8e43babc3b\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") "
Sep 19 09:09:49.354402 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config\") pod \"c13aab63-3e3a-4455-923d-2c8e43babc3b\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") "
Sep 19 09:09:49.354636 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5rt2\" (UniqueName: \"kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2\") pod \"c13aab63-3e3a-4455-923d-2c8e43babc3b\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") "
Sep 19 09:09:49.354636 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert\") pod \"c13aab63-3e3a-4455-923d-2c8e43babc3b\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") "
Sep 19 09:09:49.354636 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location\") pod \"c13aab63-3e3a-4455-923d-2c8e43babc3b\" (UID: \"c13aab63-3e3a-4455-923d-2c8e43babc3b\") "
Sep 19 09:09:49.354886 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c13aab63-3e3a-4455-923d-2c8e43babc3b" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:09:49.354886 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c13aab63-3e3a-4455-923d-2c8e43babc3b" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:09:49.355031 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.354914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c13aab63-3e3a-4455-923d-2c8e43babc3b" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:49.356569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.356542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c13aab63-3e3a-4455-923d-2c8e43babc3b" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:09:49.356680 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.356596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2" (OuterVolumeSpecName: "kube-api-access-s5rt2") pod "c13aab63-3e3a-4455-923d-2c8e43babc3b" (UID: "c13aab63-3e3a-4455-923d-2c8e43babc3b"). InnerVolumeSpecName "kube-api-access-s5rt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:49.455758 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.455717    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13aab63-3e3a-4455-923d-2c8e43babc3b-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:09:49.455758 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.455751    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13aab63-3e3a-4455-923d-2c8e43babc3b-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:09:49.455758 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.455761    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:09:49.455758 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.455782    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rt2\" (UniqueName: \"kubernetes.io/projected/c13aab63-3e3a-4455-923d-2c8e43babc3b-kube-api-access-s5rt2\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:09:49.456063 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.455792    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13aab63-3e3a-4455-923d-2c8e43babc3b-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:09:49.709932 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.709902    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:09:49.710569 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.710539    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:09:49.731778 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.731744    2566 generic.go:356] "Generic (PLEG): container finished" podID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerID="a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff" exitCode=0
Sep 19 09:09:49.731956 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.731824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerDied","Data":"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"}
Sep 19 09:09:49.731956 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.731847    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"
Sep 19 09:09:49.731956 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.731898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr" event={"ID":"c13aab63-3e3a-4455-923d-2c8e43babc3b","Type":"ContainerDied","Data":"84d462aa1c754cb3cb35899b00906ed08903485e0047d6bbec22c47b37124d89"}
Sep 19 09:09:49.731956 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.731932    2566 scope.go:117] "RemoveContainer" containerID="5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"
Sep 19 09:09:49.740405 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.740384    2566 scope.go:117] "RemoveContainer" containerID="a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"
Sep 19 09:09:49.749998 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.749973    2566 scope.go:117] "RemoveContainer" containerID="2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"
Sep 19 09:09:49.758307 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.758281    2566 scope.go:117] "RemoveContainer" containerID="7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"
Sep 19 09:09:49.766423 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.766399    2566 scope.go:117] "RemoveContainer" containerID="5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"
Sep 19 09:09:49.766748 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:09:49.766727    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827\": container with ID starting with 5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827 not found: ID does not exist" containerID="5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"
Sep 19 09:09:49.766817 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.766760    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827"} err="failed to get container status \"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827\": rpc error: code = NotFound desc = could not find container \"5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827\": container with ID starting with 5a0eb2b32ad732297e3652c73dd42001c26956506c82dfeb4f3390a0b9b2f827 not found: ID does not exist"
Sep 19 09:09:49.766817 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.766802    2566 scope.go:117] "RemoveContainer" containerID="a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"
Sep 19 09:09:49.767097 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:09:49.767077    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff\": container with ID starting with a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff not found: ID does not exist" containerID="a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"
Sep 19 09:09:49.767159 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.767104    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff"} err="failed to get container status \"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff\": rpc error: code = NotFound desc = could not find container \"a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff\": container with ID starting with a664ac1366650fea1bedd22b17d249016a309f87934ab1edc78d1a5c061b7cff not found: ID does not exist"
Sep 19 09:09:49.767159 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.767121    2566 scope.go:117] "RemoveContainer" containerID="2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"
Sep 19 09:09:49.767368 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:09:49.767349    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395\": container with ID starting with 2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395 not found: ID does not exist" containerID="2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"
Sep 19 09:09:49.767409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.767372    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395"} err="failed to get container status \"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395\": rpc error: code = NotFound desc = could not find container \"2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395\": container with ID starting with 2a8dd1642773cfb034f02ca0019d9b715588032c5018c510ac7147a6214b6395 not found: ID does not exist"
Sep 19 09:09:49.767409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.767386    2566 scope.go:117] "RemoveContainer" containerID="7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"
Sep 19 09:09:49.767632 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:09:49.767615    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc\": container with ID starting with 7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc not found: ID does not exist" containerID="7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"
Sep 19 09:09:49.767679 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.767636    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc"} err="failed to get container status \"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc\": rpc error: code = NotFound desc = could not find container \"7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc\": container with ID starting with 7c165849e1f3610dd31adfbd4f8837994c2da6be67f5f50cb857d255337b7dcc not found: ID does not exist"
Sep 19 09:09:49.775978 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.775946    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:09:49.782608 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:49.782575    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-g4ljr"]
Sep 19 09:09:50.181404 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:50.181371    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" path="/var/lib/kubelet/pods/c13aab63-3e3a-4455-923d-2c8e43babc3b/volumes"
Sep 19 09:09:51.718478 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:51.718451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:09:51.718997 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:51.718972    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:09:59.710830 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:09:59.710730    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:10:01.719297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:01.719258    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:09.710836 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:09.710794    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:10:11.719221 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:11.719178    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:19.711156 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:19.711107    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 09:10:21.719013 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:21.718958    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:29.710993 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:29.710953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:10:31.719672 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:31.719617    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:33.086439 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:33.086396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:10:35.799381 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.799340    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:10:35.799891 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.799585    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container" containerID="cri-o://3dd42b2bd8137480102ab47ab787f31a7bbcbfcc338e5b8f8dec1903872925eb" gracePeriod=30
Sep 19 09:10:35.799891 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.799639    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kube-rbac-proxy" containerID="cri-o://e0702cb86ce045af1a12b65a25bbadfc52da8dfd7dc2e24f160e20ccb06e83cd" gracePeriod=30
Sep 19 09:10:35.814796 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.814749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:10:35.919141 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.919109    2566 generic.go:356] "Generic (PLEG): container finished" podID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerID="e0702cb86ce045af1a12b65a25bbadfc52da8dfd7dc2e24f160e20ccb06e83cd" exitCode=2
Sep 19 09:10:35.919300 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:35.919179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerDied","Data":"e0702cb86ce045af1a12b65a25bbadfc52da8dfd7dc2e24f160e20ccb06e83cd"}
Sep 19 09:10:38.933699 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:38.933664    2566 generic.go:356] "Generic (PLEG): container finished" podID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerID="3dd42b2bd8137480102ab47ab787f31a7bbcbfcc338e5b8f8dec1903872925eb" exitCode=0
Sep 19 09:10:38.934088 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:38.933723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerDied","Data":"3dd42b2bd8137480102ab47ab787f31a7bbcbfcc338e5b8f8dec1903872925eb"}
Sep 19 09:10:39.033983 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.033954    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:10:39.170692 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.170646    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59qfl\" (UniqueName: \"kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl\") pod \"8611586d-27b9-4175-bbd5-2bf6e0adf718\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") "
Sep 19 09:10:39.170692 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.170700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert\") pod \"8611586d-27b9-4175-bbd5-2bf6e0adf718\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") "
Sep 19 09:10:39.170978 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.170727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config\") pod \"8611586d-27b9-4175-bbd5-2bf6e0adf718\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") "
Sep 19 09:10:39.170978 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.170815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location\") pod \"8611586d-27b9-4175-bbd5-2bf6e0adf718\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") "
Sep 19 09:10:39.170978 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.170903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls\") pod \"8611586d-27b9-4175-bbd5-2bf6e0adf718\" (UID: \"8611586d-27b9-4175-bbd5-2bf6e0adf718\") "
Sep 19 09:10:39.171207 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.171176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8611586d-27b9-4175-bbd5-2bf6e0adf718" (UID: "8611586d-27b9-4175-bbd5-2bf6e0adf718"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:10:39.171280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.171184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8611586d-27b9-4175-bbd5-2bf6e0adf718" (UID: "8611586d-27b9-4175-bbd5-2bf6e0adf718"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:10:39.172988 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.172961    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8611586d-27b9-4175-bbd5-2bf6e0adf718" (UID: "8611586d-27b9-4175-bbd5-2bf6e0adf718"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:10:39.173189 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.173169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl" (OuterVolumeSpecName: "kube-api-access-59qfl") pod "8611586d-27b9-4175-bbd5-2bf6e0adf718" (UID: "8611586d-27b9-4175-bbd5-2bf6e0adf718"). InnerVolumeSpecName "kube-api-access-59qfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:10:39.181191 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.181154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8611586d-27b9-4175-bbd5-2bf6e0adf718" (UID: "8611586d-27b9-4175-bbd5-2bf6e0adf718"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:10:39.271923 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.271879    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59qfl\" (UniqueName: \"kubernetes.io/projected/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-api-access-59qfl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:10:39.271923 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.271915    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:10:39.271923 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.271930    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8611586d-27b9-4175-bbd5-2bf6e0adf718-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:10:39.272238 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.271947    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8611586d-27b9-4175-bbd5-2bf6e0adf718-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:10:39.272238 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.271962    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8611586d-27b9-4175-bbd5-2bf6e0adf718-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:10:39.940044 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.940010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz" event={"ID":"8611586d-27b9-4175-bbd5-2bf6e0adf718","Type":"ContainerDied","Data":"5b2088d46fe830b86b5a273a0992c7858b3e82fe1b523e94b3cabc81b72b7835"}
Sep 19 09:10:39.940530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.940058    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"
Sep 19 09:10:39.940530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.940077    2566 scope.go:117] "RemoveContainer" containerID="e0702cb86ce045af1a12b65a25bbadfc52da8dfd7dc2e24f160e20ccb06e83cd"
Sep 19 09:10:39.949090 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.949071    2566 scope.go:117] "RemoveContainer" containerID="3dd42b2bd8137480102ab47ab787f31a7bbcbfcc338e5b8f8dec1903872925eb"
Sep 19 09:10:39.957466 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.957447    2566 scope.go:117] "RemoveContainer" containerID="579a95bcddc67e7425fb1d183a7ab18c405be382c38b77a9491aadad2f38f5a7"
Sep 19 09:10:39.966746 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.966722    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:10:39.971273 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:39.971249    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-8cqvz"]
Sep 19 09:10:40.180371 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.180339    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" path="/var/lib/kubelet/pods/8611586d-27b9-4175-bbd5-2bf6e0adf718/volumes"
Sep 19 09:10:40.966126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.966084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:10:40.987501 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987466    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:10:40.987752 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987738    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container"
Sep 19 09:10:40.987752 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987761    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987779    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987787    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="storage-initializer"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="storage-initializer"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-agent"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987821    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-agent"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kube-rbac-proxy"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987833    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kube-rbac-proxy"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987845    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="storage-initializer"
Sep 19 09:10:40.987864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987850    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="storage-initializer"
Sep 19 09:10:40.988266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987893    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kube-rbac-proxy"
Sep 19 09:10:40.988266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987902    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-container"
Sep 19 09:10:40.988266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987908    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kube-rbac-proxy"
Sep 19 09:10:40.988266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987913    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8611586d-27b9-4175-bbd5-2bf6e0adf718" containerName="kserve-container"
Sep 19 09:10:40.988266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.987921    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13aab63-3e3a-4455-923d-2c8e43babc3b" containerName="kserve-agent"
Sep 19 09:10:40.990981 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.990961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:40.993481 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.993459    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:10:40.993711 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.993691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:10:40.993821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:40.993744    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 19 09:10:41.002348 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.002323    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:10:41.086083 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.086040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.086274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.086096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.086274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.086184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.086274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.086216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxw8\" (UniqueName: \"kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.086274 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.086242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187199 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxw8\" (UniqueName: \"kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.187841 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.187820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.188041 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.188022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.188085 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.188065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.189644 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.189622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.196476 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.196452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxw8\" (UniqueName: \"kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crp55\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.205722 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.205687    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:10:41.300892 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.300796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:41.574358 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.574175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:10:41.576391 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:10:41.576365    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc858eb0_be61_43af_8089_05a2d9848c97.slice/crio-91651eaeb01c546e362ab0dcb104c52ce5e4f46caa7d9b9ebf2397de90011244 WatchSource:0}: Error finding container 91651eaeb01c546e362ab0dcb104c52ce5e4f46caa7d9b9ebf2397de90011244: Status 404 returned error can't find the container with id 91651eaeb01c546e362ab0dcb104c52ce5e4f46caa7d9b9ebf2397de90011244
Sep 19 09:10:41.719672 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.719627    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:41.957947 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.957898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerStarted","Data":"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"}
Sep 19 09:10:41.958150 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:41.958020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerStarted","Data":"91651eaeb01c546e362ab0dcb104c52ce5e4f46caa7d9b9ebf2397de90011244"}
Sep 19 09:10:46.981490 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:46.981453    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc858eb0-be61-43af-8089-05a2d9848c97" containerID="636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6" exitCode=0
Sep 19 09:10:46.982015 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:46.981528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerDied","Data":"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"}
Sep 19 09:10:47.987806 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:47.987756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerStarted","Data":"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"}
Sep 19 09:10:47.988227 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:47.987839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerStarted","Data":"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"}
Sep 19 09:10:47.988227 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:47.988125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:47.988311 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:47.988279    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:47.989578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:47.989551    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:10:48.014726 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:48.014670    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podStartSLOduration=3.40011743 podStartE2EDuration="8.01465161s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.61453418s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:10:48.012266388 +0000 UTC m=+3012.332937991" watchObservedRunningTime="2026-09-19 09:10:48.01465161 +0000 UTC m=+3012.335323199"
Sep 19 09:10:48.991485 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:48.991443    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:10:51.719482 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:51.719434    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:10:54.000638 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:54.000604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:10:54.001163 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:10:54.001136    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:11:01.719600 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:01.719561    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:11:04.001957 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:04.001915    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:11:11.719939 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:11.719908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:11:14.002000 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:14.001960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:11:17.853195 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:17.853160    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:11:17.854794 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:17.853415    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" containerID="cri-o://efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f" gracePeriod=30
Sep 19 09:11:17.854794 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:17.853482    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kube-rbac-proxy" containerID="cri-o://c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792" gracePeriod=30
Sep 19 09:11:17.879201 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:17.879174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:18.110620 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:18.110593    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerID="c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792" exitCode=2
Sep 19 09:11:18.110762 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:18.110666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerDied","Data":"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"}
Sep 19 09:11:18.930402 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:18.930365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:19.126218 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:19.126186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:21.713977 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:21.713938    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 19 09:11:21.719251 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:21.719231    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 19 09:11:22.287750 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.287721    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:11:22.404905 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.404828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfnvf\" (UniqueName: \"kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf\") pod \"bf168c2a-443a-42bc-9207-e8a75e966b79\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") "
Sep 19 09:11:22.404905 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.404871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location\") pod \"bf168c2a-443a-42bc-9207-e8a75e966b79\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") "
Sep 19 09:11:22.405172 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.404906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert\") pod \"bf168c2a-443a-42bc-9207-e8a75e966b79\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") "
Sep 19 09:11:22.405172 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.404932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls\") pod \"bf168c2a-443a-42bc-9207-e8a75e966b79\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") "
Sep 19 09:11:22.405172 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.404994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config\") pod \"bf168c2a-443a-42bc-9207-e8a75e966b79\" (UID: \"bf168c2a-443a-42bc-9207-e8a75e966b79\") "
Sep 19 09:11:22.405313 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.405244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf168c2a-443a-42bc-9207-e8a75e966b79" (UID: "bf168c2a-443a-42bc-9207-e8a75e966b79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:22.405362 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.405282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf168c2a-443a-42bc-9207-e8a75e966b79" (UID: "bf168c2a-443a-42bc-9207-e8a75e966b79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:22.405429 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.405407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf168c2a-443a-42bc-9207-e8a75e966b79" (UID: "bf168c2a-443a-42bc-9207-e8a75e966b79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:22.406805 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.406782    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf" (OuterVolumeSpecName: "kube-api-access-rfnvf") pod "bf168c2a-443a-42bc-9207-e8a75e966b79" (UID: "bf168c2a-443a-42bc-9207-e8a75e966b79"). InnerVolumeSpecName "kube-api-access-rfnvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:11:22.406971 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.406951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf168c2a-443a-42bc-9207-e8a75e966b79" (UID: "bf168c2a-443a-42bc-9207-e8a75e966b79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:11:22.506343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.506312    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnvf\" (UniqueName: \"kubernetes.io/projected/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-api-access-rfnvf\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:22.506343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.506339    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf168c2a-443a-42bc-9207-e8a75e966b79-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:22.506343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.506348    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:22.506555 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.506357    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf168c2a-443a-42bc-9207-e8a75e966b79-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:22.506555 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:22.506365    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf168c2a-443a-42bc-9207-e8a75e966b79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:23.130526 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.130495    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerID="efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f" exitCode=0
Sep 19 09:11:23.130899 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.130566    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"
Sep 19 09:11:23.130899 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.130577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerDied","Data":"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"}
Sep 19 09:11:23.130899 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.130665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9" event={"ID":"bf168c2a-443a-42bc-9207-e8a75e966b79","Type":"ContainerDied","Data":"f0b999c2a93eb48892edafbc4bdcc0e5b40a468f4ee8c21bec3ddb30b936d93c"}
Sep 19 09:11:23.130899 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.130695    2566 scope.go:117] "RemoveContainer" containerID="c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"
Sep 19 09:11:23.139516 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.139498    2566 scope.go:117] "RemoveContainer" containerID="efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"
Sep 19 09:11:23.147798 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.147760    2566 scope.go:117] "RemoveContainer" containerID="b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"
Sep 19 09:11:23.153259 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.153237    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:11:23.156493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.156471    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-mzcg9"]
Sep 19 09:11:23.156943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.156926    2566 scope.go:117] "RemoveContainer" containerID="c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"
Sep 19 09:11:23.157220 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:23.157201    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792\": container with ID starting with c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792 not found: ID does not exist" containerID="c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"
Sep 19 09:11:23.157277 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.157231    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792"} err="failed to get container status \"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792\": rpc error: code = NotFound desc = could not find container \"c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792\": container with ID starting with c98a0c1b4757fa3751d29f6e1ec5e2f4f15ecb25d07dc476a387a693b13ad792 not found: ID does not exist"
Sep 19 09:11:23.157277 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.157248    2566 scope.go:117] "RemoveContainer" containerID="efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"
Sep 19 09:11:23.157518 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:23.157504    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f\": container with ID starting with efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f not found: ID does not exist" containerID="efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"
Sep 19 09:11:23.157561 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.157525    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f"} err="failed to get container status \"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f\": rpc error: code = NotFound desc = could not find container \"efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f\": container with ID starting with efb73b8c49010b8f3805bc3ceb0b6b15532e0425e34f7f81fb92231d92bea87f not found: ID does not exist"
Sep 19 09:11:23.157561 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.157549    2566 scope.go:117] "RemoveContainer" containerID="b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"
Sep 19 09:11:23.157883 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:23.157853    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e\": container with ID starting with b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e not found: ID does not exist" containerID="b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"
Sep 19 09:11:23.158000 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:23.157889    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e"} err="failed to get container status \"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e\": rpc error: code = NotFound desc = could not find container \"b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e\": container with ID starting with b7cc06d159f94ef28aeff37e892103a8091e963bfafbd62a086a30a86ce0cb2e not found: ID does not exist"
Sep 19 09:11:24.001551 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:24.001512    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:11:24.180482 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:24.180451    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" path="/var/lib/kubelet/pods/bf168c2a-443a-42bc-9207-e8a75e966b79/volumes"
Sep 19 09:11:28.964659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:28.964584    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:30.060824 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:30.060783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:30.610740 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:30.610697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:34.001867 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:34.001833    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:11:42.386314 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:42.386282    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:11:42.386694 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:42.386534    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" containerID="cri-o://676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618" gracePeriod=30
Sep 19 09:11:42.386694 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:42.386553    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kube-rbac-proxy" containerID="cri-o://42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000" gracePeriod=30
Sep 19 09:11:42.415617 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:42.415560    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:43.212550 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:43.212513    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc858eb0-be61-43af-8089-05a2d9848c97" containerID="42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000" exitCode=2
Sep 19 09:11:43.212747 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:43.212592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerDied","Data":"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"}
Sep 19 09:11:43.992213 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:43.992166    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 19 09:11:44.002006 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:44.001978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 19 09:11:45.026008 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.025986    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:11:45.180203 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hxw8\" (UniqueName: \"kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8\") pod \"dc858eb0-be61-43af-8089-05a2d9848c97\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") "
Sep 19 09:11:45.180351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config\") pod \"dc858eb0-be61-43af-8089-05a2d9848c97\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") "
Sep 19 09:11:45.180351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls\") pod \"dc858eb0-be61-43af-8089-05a2d9848c97\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") "
Sep 19 09:11:45.180351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert\") pod \"dc858eb0-be61-43af-8089-05a2d9848c97\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") "
Sep 19 09:11:45.180351 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location\") pod \"dc858eb0-be61-43af-8089-05a2d9848c97\" (UID: \"dc858eb0-be61-43af-8089-05a2d9848c97\") "
Sep 19 09:11:45.180706 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180675    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc858eb0-be61-43af-8089-05a2d9848c97" (UID: "dc858eb0-be61-43af-8089-05a2d9848c97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:45.180795 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.180692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc858eb0-be61-43af-8089-05a2d9848c97" (UID: "dc858eb0-be61-43af-8089-05a2d9848c97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:45.182373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.182348    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc858eb0-be61-43af-8089-05a2d9848c97" (UID: "dc858eb0-be61-43af-8089-05a2d9848c97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:11:45.182373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.182365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8" (OuterVolumeSpecName: "kube-api-access-2hxw8") pod "dc858eb0-be61-43af-8089-05a2d9848c97" (UID: "dc858eb0-be61-43af-8089-05a2d9848c97"). InnerVolumeSpecName "kube-api-access-2hxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:11:45.191173 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.191148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc858eb0-be61-43af-8089-05a2d9848c97" (UID: "dc858eb0-be61-43af-8089-05a2d9848c97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:45.222335 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.222312    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc858eb0-be61-43af-8089-05a2d9848c97" containerID="676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618" exitCode=0
Sep 19 09:11:45.222403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.222363    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"
Sep 19 09:11:45.222403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.222373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerDied","Data":"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"}
Sep 19 09:11:45.222483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.222419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55" event={"ID":"dc858eb0-be61-43af-8089-05a2d9848c97","Type":"ContainerDied","Data":"91651eaeb01c546e362ab0dcb104c52ce5e4f46caa7d9b9ebf2397de90011244"}
Sep 19 09:11:45.222483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.222448    2566 scope.go:117] "RemoveContainer" containerID="42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"
Sep 19 09:11:45.230902 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.230879    2566 scope.go:117] "RemoveContainer" containerID="676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"
Sep 19 09:11:45.237713 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.237698    2566 scope.go:117] "RemoveContainer" containerID="636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"
Sep 19 09:11:45.244914 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.244890    2566 scope.go:117] "RemoveContainer" containerID="42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"
Sep 19 09:11:45.245003 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.244986    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:11:45.245166 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:45.245149    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000\": container with ID starting with 42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000 not found: ID does not exist" containerID="42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"
Sep 19 09:11:45.245217 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.245173    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000"} err="failed to get container status \"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000\": rpc error: code = NotFound desc = could not find container \"42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000\": container with ID starting with 42d79dcdd3230a2d9c63a17c06a7127aca504118f4c061e5acb1e047824ca000 not found: ID does not exist"
Sep 19 09:11:45.245217 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.245190    2566 scope.go:117] "RemoveContainer" containerID="676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"
Sep 19 09:11:45.245440 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:45.245423    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618\": container with ID starting with 676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618 not found: ID does not exist" containerID="676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"
Sep 19 09:11:45.245478 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.245446    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618"} err="failed to get container status \"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618\": rpc error: code = NotFound desc = could not find container \"676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618\": container with ID starting with 676f9e23044d3ced792408a75d627027776993dec1eb80b4cbd3b2c09eb33618 not found: ID does not exist"
Sep 19 09:11:45.245478 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.245462    2566 scope.go:117] "RemoveContainer" containerID="636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"
Sep 19 09:11:45.245679 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:45.245665    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6\": container with ID starting with 636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6 not found: ID does not exist" containerID="636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"
Sep 19 09:11:45.245715 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.245683    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6"} err="failed to get container status \"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6\": rpc error: code = NotFound desc = could not find container \"636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6\": container with ID starting with 636a84245d5e416335b76618eea26c1193718cfe4f93f504b8898531db4068a6 not found: ID does not exist"
Sep 19 09:11:45.249488 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.249467    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crp55"]
Sep 19 09:11:45.281452 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.281434    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxw8\" (UniqueName: \"kubernetes.io/projected/dc858eb0-be61-43af-8089-05a2d9848c97-kube-api-access-2hxw8\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:45.281544 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.281453    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:45.281544 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.281465    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc858eb0-be61-43af-8089-05a2d9848c97-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:45.281544 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.281474    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc858eb0-be61-43af-8089-05a2d9848c97-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:45.281544 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:45.281483    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc858eb0-be61-43af-8089-05a2d9848c97-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:11:46.180690 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:46.180656    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" path="/var/lib/kubelet/pods/dc858eb0-be61-43af-8089-05a2d9848c97/volumes"
Sep 19 09:11:47.536301 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.536264    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:47.566431 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566399    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:11:47.566711 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566695    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kube-rbac-proxy"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566715    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kube-rbac-proxy"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566732    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566742    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="storage-initializer"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="storage-initializer"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566793    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566801    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="storage-initializer"
Sep 19 09:11:47.566816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566817    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="storage-initializer"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566836    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kube-rbac-proxy"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566866    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kube-rbac-proxy"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566939    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kserve-container"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566952    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kube-rbac-proxy"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc858eb0-be61-43af-8089-05a2d9848c97" containerName="kube-rbac-proxy"
Sep 19 09:11:47.567262 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.566971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf168c2a-443a-42bc-9207-e8a75e966b79" containerName="kserve-container"
Sep 19 09:11:47.570528 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.570508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.572862 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.572824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:11:47.572982 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.572879    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 19 09:11:47.573189 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.573172    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 19 09:11:47.573276 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.573248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:11:47.573950 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.573932    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:11:47.574040 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.573946    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:11:47.574040 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.573935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:11:47.580301 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.580280    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:11:47.697986 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.697963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.698089 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.697996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.698089 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.698024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdmf\" (UniqueName: \"kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.698089 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.698076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.698197 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.698101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798867 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdmf\" (UniqueName: \"kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798867 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.798867 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:47.798751    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 19 09:11:47.798867 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:11:47.798842    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls podName:457605a8-376c-407f-bf27-bdcda5d2ca25 nodeName:}" failed. No retries permitted until 2026-09-19 09:11:48.298821838 +0000 UTC m=+3072.619493420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 19 09:11:47.799068 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.798993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.799153 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.799136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.799252 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.799235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:47.807649 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:47.807631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdmf\" (UniqueName: \"kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:48.020836 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:48.020803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:11:48.302096 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:48.302067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:48.304364 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:48.304335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:48.481143 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:48.481115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:48.750361 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:48.750336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:11:48.752908 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:11:48.752875    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457605a8_376c_407f_bf27_bdcda5d2ca25.slice/crio-9c3821ed5ceca81149031f33c7d5a7883d95109d23e701e76502749c42cafc12 WatchSource:0}: Error finding container 9c3821ed5ceca81149031f33c7d5a7883d95109d23e701e76502749c42cafc12: Status 404 returned error can't find the container with id 9c3821ed5ceca81149031f33c7d5a7883d95109d23e701e76502749c42cafc12
Sep 19 09:11:49.238892 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:49.238856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerStarted","Data":"2f3e1fd736e7b9fb7b9d17d43afe54429bda12d42bc3b0fb1f6c0c7874693329"}
Sep 19 09:11:49.239058 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:49.238935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerStarted","Data":"9c3821ed5ceca81149031f33c7d5a7883d95109d23e701e76502749c42cafc12"}
Sep 19 09:11:53.255478 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:53.255449    2566 generic.go:356] "Generic (PLEG): container finished" podID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerID="2f3e1fd736e7b9fb7b9d17d43afe54429bda12d42bc3b0fb1f6c0c7874693329" exitCode=0
Sep 19 09:11:53.255789 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:53.255493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerDied","Data":"2f3e1fd736e7b9fb7b9d17d43afe54429bda12d42bc3b0fb1f6c0c7874693329"}
Sep 19 09:11:54.266349 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:54.266312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerStarted","Data":"ab519d9a3ba5b9d380ad42ef802f4265b778858cf19510f0105c69ebf0f1032d"}
Sep 19 09:11:54.266688 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:54.266400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerStarted","Data":"836ee2f48bb4eb10eb731ff630896c5752b7a8437f45b13e63c493e6e07a7c29"}
Sep 19 09:11:54.266688 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:54.266582    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:54.288252 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:54.288209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podStartSLOduration=2.896556523 podStartE2EDuration="7.288195832s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.391639309s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:11:54.286482369 +0000 UTC m=+3078.607153956" watchObservedRunningTime="2026-09-19 09:11:54.288195832 +0000 UTC m=+3078.608867420"
Sep 19 09:11:55.270754 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:55.270721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:11:55.272039 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:55.272011    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:11:56.274532 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:11:56.274490    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:01.278663 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:01.278632    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:12:01.279250 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:01.279223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:11.279408 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:11.279367    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:14.966204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:14.966163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:12:15.065498 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:15.065461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:12:21.279915 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:21.279874    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:31.279251 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:31.279215    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:41.280695 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:41.280657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:12:49.299243 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.299207    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:12:49.299689 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.299448    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" containerID="cri-o://836ee2f48bb4eb10eb731ff630896c5752b7a8437f45b13e63c493e6e07a7c29" gracePeriod=30
Sep 19 09:12:49.299689 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.299531    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kube-rbac-proxy" containerID="cri-o://ab519d9a3ba5b9d380ad42ef802f4265b778858cf19510f0105c69ebf0f1032d" gracePeriod=30
Sep 19 09:12:49.316695 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.316661    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:12:49.471928 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.471896    2566 generic.go:356] "Generic (PLEG): container finished" podID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerID="ab519d9a3ba5b9d380ad42ef802f4265b778858cf19510f0105c69ebf0f1032d" exitCode=2
Sep 19 09:12:49.472103 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:49.471951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerDied","Data":"ab519d9a3ba5b9d380ad42ef802f4265b778858cf19510f0105c69ebf0f1032d"}
Sep 19 09:12:51.275369 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:51.275319    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 19 09:12:51.279691 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:51.279660    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 19 09:12:52.487421 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.487392    2566 generic.go:356] "Generic (PLEG): container finished" podID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerID="836ee2f48bb4eb10eb731ff630896c5752b7a8437f45b13e63c493e6e07a7c29" exitCode=0
Sep 19 09:12:52.488083 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.487462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerDied","Data":"836ee2f48bb4eb10eb731ff630896c5752b7a8437f45b13e63c493e6e07a7c29"}
Sep 19 09:12:52.539578 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.539551    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:12:52.672278 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config\") pod \"457605a8-376c-407f-bf27-bdcda5d2ca25\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") "
Sep 19 09:12:52.672465 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert\") pod \"457605a8-376c-407f-bf27-bdcda5d2ca25\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") "
Sep 19 09:12:52.672465 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672319    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location\") pod \"457605a8-376c-407f-bf27-bdcda5d2ca25\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") "
Sep 19 09:12:52.672465 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672338    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzdmf\" (UniqueName: \"kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf\") pod \"457605a8-376c-407f-bf27-bdcda5d2ca25\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") "
Sep 19 09:12:52.672465 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") pod \"457605a8-376c-407f-bf27-bdcda5d2ca25\" (UID: \"457605a8-376c-407f-bf27-bdcda5d2ca25\") "
Sep 19 09:12:52.672716 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "457605a8-376c-407f-bf27-bdcda5d2ca25" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:12:52.672798 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.672689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "457605a8-376c-407f-bf27-bdcda5d2ca25" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:12:52.674538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.674504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "457605a8-376c-407f-bf27-bdcda5d2ca25" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:12:52.674658 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.674583    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf" (OuterVolumeSpecName: "kube-api-access-mzdmf") pod "457605a8-376c-407f-bf27-bdcda5d2ca25" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25"). InnerVolumeSpecName "kube-api-access-mzdmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:12:52.682436 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.682408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "457605a8-376c-407f-bf27-bdcda5d2ca25" (UID: "457605a8-376c-407f-bf27-bdcda5d2ca25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:52.773073 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.773031    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:12:52.773073 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.773068    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/457605a8-376c-407f-bf27-bdcda5d2ca25-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:12:52.773073 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.773078    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457605a8-376c-407f-bf27-bdcda5d2ca25-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:12:52.773073 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.773086    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdmf\" (UniqueName: \"kubernetes.io/projected/457605a8-376c-407f-bf27-bdcda5d2ca25-kube-api-access-mzdmf\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:12:52.773352 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:52.773096    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/457605a8-376c-407f-bf27-bdcda5d2ca25-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:12:53.493406 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.493370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx" event={"ID":"457605a8-376c-407f-bf27-bdcda5d2ca25","Type":"ContainerDied","Data":"9c3821ed5ceca81149031f33c7d5a7883d95109d23e701e76502749c42cafc12"}
Sep 19 09:12:53.493406 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.493408    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"
Sep 19 09:12:53.493983 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.493437    2566 scope.go:117] "RemoveContainer" containerID="ab519d9a3ba5b9d380ad42ef802f4265b778858cf19510f0105c69ebf0f1032d"
Sep 19 09:12:53.505052 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.505001    2566 scope.go:117] "RemoveContainer" containerID="836ee2f48bb4eb10eb731ff630896c5752b7a8437f45b13e63c493e6e07a7c29"
Sep 19 09:12:53.513050 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.513030    2566 scope.go:117] "RemoveContainer" containerID="2f3e1fd736e7b9fb7b9d17d43afe54429bda12d42bc3b0fb1f6c0c7874693329"
Sep 19 09:12:53.521265 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.521242    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:12:53.525656 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:53.525634    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-dbgkx"]
Sep 19 09:12:54.180367 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.180333    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" path="/var/lib/kubelet/pods/457605a8-376c-407f-bf27-bdcda5d2ca25/volumes"
Sep 19 09:12:54.474415 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.474324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:12:54.526883 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.526846    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527110    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kube-rbac-proxy"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527120    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kube-rbac-proxy"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527137    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="storage-initializer"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527144    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="storage-initializer"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527153    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527158    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527197    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kube-rbac-proxy"
Sep 19 09:12:54.527280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.527205    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="457605a8-376c-407f-bf27-bdcda5d2ca25" containerName="kserve-container"
Sep 19 09:12:54.531700 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.531682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.534196 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.534170    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:12:54.534313 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.534216    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:12:54.534313 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.534217    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:12:54.534538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.534521    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:12:54.535074 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.535058    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 19 09:12:54.535196 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.535179    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:12:54.535288 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.535204    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 19 09:12:54.545508 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.545486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:12:54.587659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.587623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.587837 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.587687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgnm\" (UniqueName: \"kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.587837 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.587717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.587837 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.587733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.587837 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.587754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.599637 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.599597    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:12:54.689154 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgnm\" (UniqueName: \"kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.689331 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.689331 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.689331 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.689331 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.689679 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.689656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.690023 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.690003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.690023 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.690017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.691782 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.691754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.698831 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.698808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgnm\" (UniqueName: \"kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm\") pod \"isvc-pmml-predictor-8674795f59-dzhd6\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:54.841495 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:54.841398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:12:55.116759 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:55.116730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:12:55.119261 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:12:55.119235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec7012c_a243_4beb_9e2f_006196c53b47.slice/crio-b259c5c5c7580685e107651c45ffc28237b40aef86c04db12cc899e1bd8e4fde WatchSource:0}: Error finding container b259c5c5c7580685e107651c45ffc28237b40aef86c04db12cc899e1bd8e4fde: Status 404 returned error can't find the container with id b259c5c5c7580685e107651c45ffc28237b40aef86c04db12cc899e1bd8e4fde
Sep 19 09:12:55.504291 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:55.504212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerStarted","Data":"2e068caa87e1cdb6421afd62dba613e05a78ecd143da669634d704ebcaa382a7"}
Sep 19 09:12:55.504291 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:55.504268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerStarted","Data":"b259c5c5c7580685e107651c45ffc28237b40aef86c04db12cc899e1bd8e4fde"}
Sep 19 09:12:59.520753 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:59.520666    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerID="2e068caa87e1cdb6421afd62dba613e05a78ecd143da669634d704ebcaa382a7" exitCode=0
Sep 19 09:12:59.520753 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:12:59.520720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerDied","Data":"2e068caa87e1cdb6421afd62dba613e05a78ecd143da669634d704ebcaa382a7"}
Sep 19 09:13:07.562113 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:07.562076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerStarted","Data":"a3282362bc773a7296ecc0d6f7592a5f62d190a4bc1338a48a2027f25d827d48"}
Sep 19 09:13:07.562499 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:07.562140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerStarted","Data":"3f5c8a46ba790c5d0d4d6680135e4ee1d30ab171ac8d30af33cccf4a004ec7f2"}
Sep 19 09:13:07.562499 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:07.562318    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:13:07.582755 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:07.582704    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podStartSLOduration=2.28317182 podStartE2EDuration="13.582686642s" totalImagesPullingTime="7.329678862s" totalInitContainerRuntime="3.96983596s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:13:07.580804059 +0000 UTC m=+3151.901475637" watchObservedRunningTime="2026-09-19 09:13:07.582686642 +0000 UTC m=+3151.903358230"
Sep 19 09:13:08.566088 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:08.566052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:13:08.567326 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:08.567297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:09.570312 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:09.570276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:10.115848 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:10.115806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:11.203424 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:11.203389    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:11.374011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:11.373980    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:14.575088 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:14.575059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:13:14.575572 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:14.575543    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:21.200439 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:21.200400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:22.296381 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:22.296348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:23.230567 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:23.230529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:13:24.576502 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:24.576465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:34.575625 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:34.575575    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:44.575747 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:44.575706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:13:54.575565 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:13:54.575522    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:14:04.576202 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:04.576162    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:14:14.576015 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:14.575975    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:14:21.176167 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:21.176129    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:14:31.176104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:31.176011    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 19 09:14:41.176904 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:41.176860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:14:46.082839 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.082806    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:14:46.083191 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.083052    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container" containerID="cri-o://3f5c8a46ba790c5d0d4d6680135e4ee1d30ab171ac8d30af33cccf4a004ec7f2" gracePeriod=30
Sep 19 09:14:46.083191 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.083082    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kube-rbac-proxy" containerID="cri-o://a3282362bc773a7296ecc0d6f7592a5f62d190a4bc1338a48a2027f25d827d48" gracePeriod=30
Sep 19 09:14:46.109598 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.109572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:14:46.937901 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.937873    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerID="a3282362bc773a7296ecc0d6f7592a5f62d190a4bc1338a48a2027f25d827d48" exitCode=2
Sep 19 09:14:46.938051 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:46.937920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerDied","Data":"a3282362bc773a7296ecc0d6f7592a5f62d190a4bc1338a48a2027f25d827d48"}
Sep 19 09:14:49.570878 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:49.570839    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 19 09:14:49.954090 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:49.954062    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerID="3f5c8a46ba790c5d0d4d6680135e4ee1d30ab171ac8d30af33cccf4a004ec7f2" exitCode=0
Sep 19 09:14:49.954228 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:49.954129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerDied","Data":"3f5c8a46ba790c5d0d4d6680135e4ee1d30ab171ac8d30af33cccf4a004ec7f2"}
Sep 19 09:14:50.013149 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.013127    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:14:50.099050 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.098968    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvgnm\" (UniqueName: \"kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm\") pod \"8ec7012c-a243-4beb-9e2f-006196c53b47\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") "
Sep 19 09:14:50.099050 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location\") pod \"8ec7012c-a243-4beb-9e2f-006196c53b47\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") "
Sep 19 09:14:50.099050 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert\") pod \"8ec7012c-a243-4beb-9e2f-006196c53b47\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") "
Sep 19 09:14:50.099308 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls\") pod \"8ec7012c-a243-4beb-9e2f-006196c53b47\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") "
Sep 19 09:14:50.099308 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config\") pod \"8ec7012c-a243-4beb-9e2f-006196c53b47\" (UID: \"8ec7012c-a243-4beb-9e2f-006196c53b47\") "
Sep 19 09:14:50.099421 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ec7012c-a243-4beb-9e2f-006196c53b47" (UID: "8ec7012c-a243-4beb-9e2f-006196c53b47"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:14:50.099421 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ec7012c-a243-4beb-9e2f-006196c53b47" (UID: "8ec7012c-a243-4beb-9e2f-006196c53b47"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:50.099546 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.099519    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ec7012c-a243-4beb-9e2f-006196c53b47" (UID: "8ec7012c-a243-4beb-9e2f-006196c53b47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:14:50.101108 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.101076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm" (OuterVolumeSpecName: "kube-api-access-fvgnm") pod "8ec7012c-a243-4beb-9e2f-006196c53b47" (UID: "8ec7012c-a243-4beb-9e2f-006196c53b47"). InnerVolumeSpecName "kube-api-access-fvgnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:14:50.101194 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.101162    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ec7012c-a243-4beb-9e2f-006196c53b47" (UID: "8ec7012c-a243-4beb-9e2f-006196c53b47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:14:50.200491 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.200470    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:14:50.200491 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.200493    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ec7012c-a243-4beb-9e2f-006196c53b47-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:14:50.200638 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.200502    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:14:50.200638 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.200511    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgnm\" (UniqueName: \"kubernetes.io/projected/8ec7012c-a243-4beb-9e2f-006196c53b47-kube-api-access-fvgnm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:14:50.200638 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.200520    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ec7012c-a243-4beb-9e2f-006196c53b47-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:14:50.959713 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.959677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6" event={"ID":"8ec7012c-a243-4beb-9e2f-006196c53b47","Type":"ContainerDied","Data":"b259c5c5c7580685e107651c45ffc28237b40aef86c04db12cc899e1bd8e4fde"}
Sep 19 09:14:50.960127 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.959749    2566 scope.go:117] "RemoveContainer" containerID="a3282362bc773a7296ecc0d6f7592a5f62d190a4bc1338a48a2027f25d827d48"
Sep 19 09:14:50.960127 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.959690    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"
Sep 19 09:14:50.967676 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.967656    2566 scope.go:117] "RemoveContainer" containerID="3f5c8a46ba790c5d0d4d6680135e4ee1d30ab171ac8d30af33cccf4a004ec7f2"
Sep 19 09:14:50.974629 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.974613    2566 scope.go:117] "RemoveContainer" containerID="2e068caa87e1cdb6421afd62dba613e05a78ecd143da669634d704ebcaa382a7"
Sep 19 09:14:50.978719 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.978700    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:14:50.982734 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:50.982716    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dzhd6"]
Sep 19 09:14:51.269030 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.268999    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:14:51.269254 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269242    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kube-rbac-proxy"
Sep 19 09:14:51.269297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269256    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kube-rbac-proxy"
Sep 19 09:14:51.269297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269276    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container"
Sep 19 09:14:51.269297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269281    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container"
Sep 19 09:14:51.269297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269289    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="storage-initializer"
Sep 19 09:14:51.269297 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269296    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="storage-initializer"
Sep 19 09:14:51.269473 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269335    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kube-rbac-proxy"
Sep 19 09:14:51.269473 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.269342    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" containerName="kserve-container"
Sep 19 09:14:51.273490 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.273474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.276018 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.275993    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:14:51.276138 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276113    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 19 09:14:51.276257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276220    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:14:51.276257 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276243    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:14:51.276373 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276326    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:14:51.276622 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276608    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:14:51.276994 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.276979    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:14:51.282498 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.282478    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:14:51.307721 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.307696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsgp\" (UniqueName: \"kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.307833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.307731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.307833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.307757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.307833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.307811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.307833 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.307830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.408943 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.408914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsgp\" (UniqueName: \"kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.408961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.408998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.409027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.409051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409284 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:14:51.409171    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 19 09:14:51.409284 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:14:51.409233    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls podName:db10bd12-6e1f-4242-8c5b-a4a16488c0e4 nodeName:}" failed. No retries permitted until 2026-09-19 09:14:51.909213535 +0000 UTC m=+3256.229885120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-xvvxz" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 19 09:14:51.409403 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.409362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409700 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.409680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.409795 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.409684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.418501 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.418483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsgp\" (UniqueName: \"kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.913873 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.913838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.916067 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.916040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xvvxz\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:51.950917 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:51.950888    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:14:52.013011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.012982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:14:52.180630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.180571    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec7012c-a243-4beb-9e2f-006196c53b47" path="/var/lib/kubelet/pods/8ec7012c-a243-4beb-9e2f-006196c53b47/volumes"
Sep 19 09:14:52.183557 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.183539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:52.456739 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.456718    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:14:52.458829 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:14:52.458803    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb10bd12_6e1f_4242_8c5b_a4a16488c0e4.slice/crio-fa4048a51ffe36b08739afcfef5ac23111c05c85b311e0e85408639aeaf8213c WatchSource:0}: Error finding container fa4048a51ffe36b08739afcfef5ac23111c05c85b311e0e85408639aeaf8213c: Status 404 returned error can't find the container with id fa4048a51ffe36b08739afcfef5ac23111c05c85b311e0e85408639aeaf8213c
Sep 19 09:14:52.970904 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.970866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerStarted","Data":"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"}
Sep 19 09:14:52.971100 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:52.970923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerStarted","Data":"fa4048a51ffe36b08739afcfef5ac23111c05c85b311e0e85408639aeaf8213c"}
Sep 19 09:14:56.986635 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:56.986604    2566 generic.go:356] "Generic (PLEG): container finished" podID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerID="50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd" exitCode=0
Sep 19 09:14:56.986635 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:56.986638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerDied","Data":"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"}
Sep 19 09:14:57.993370 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:57.993337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerStarted","Data":"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"}
Sep 19 09:14:57.993786 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:57.993398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerStarted","Data":"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"}
Sep 19 09:14:57.993786 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:57.993589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:58.015284 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:58.015237    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podStartSLOduration=3.366632586 podStartE2EDuration="7.015224703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648592117s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:14:58.014285646 +0000 UTC m=+3262.334957274" watchObservedRunningTime="2026-09-19 09:14:58.015224703 +0000 UTC m=+3262.335896290"
Sep 19 09:14:58.998321 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:58.998286    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:14:58.999390 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:14:58.999364    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:00.002397 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:00.002363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:02.352182 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:02.352145    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:03.422822 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.422792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:03.439500 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.439470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:03.442835 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.442818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.446068 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.446041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 19 09:15:03.446212 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.446079    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 19 09:15:03.446212 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.446051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:15:03.459958 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.459931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:03.496702 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.496672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.496856 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.496705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6c8\" (UniqueName: \"kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.496856 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.496791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.496856 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.496844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598120 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6c8\" (UniqueName: \"kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598343 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:15:03.598240    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 19 09:15:03.598343 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:15:03.598300    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls podName:c97f006c-afb7-4464-a6be-e003dd680fe6 nodeName:}" failed. No retries permitted until 2026-09-19 09:15:04.098284335 +0000 UTC m=+3268.418955924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" (UID: "c97f006c-afb7-4464-a6be-e003dd680fe6") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 19 09:15:03.598631 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.598909 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.598893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.609541 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.609518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6c8\" (UniqueName: \"kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:03.903321 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:03.903283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:04.102510 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:04.102474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:04.104779 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:04.104744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:04.352904 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:04.352828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:04.621595 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:04.621569    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:04.624300 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:15:04.624267    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97f006c_afb7_4464_a6be_e003dd680fe6.slice/crio-0bc3c67d2aa144f2eff53960489709cffda3db99a017861587d145f0fece5eee WatchSource:0}: Error finding container 0bc3c67d2aa144f2eff53960489709cffda3db99a017861587d145f0fece5eee: Status 404 returned error can't find the container with id 0bc3c67d2aa144f2eff53960489709cffda3db99a017861587d145f0fece5eee
Sep 19 09:15:05.006760 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:05.006689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:15:05.007317 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:05.007292    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:05.023517 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:05.023487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerStarted","Data":"69133d3632bcafa59cb764647713a7fc1a968abab529d60eb73a7d01ae35e503"}
Sep 19 09:15:05.023653 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:05.023546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerStarted","Data":"0bc3c67d2aa144f2eff53960489709cffda3db99a017861587d145f0fece5eee"}
Sep 19 09:15:11.049496 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:11.049473    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/0.log"
Sep 19 09:15:11.049850 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:11.049511    2566 generic.go:356] "Generic (PLEG): container finished" podID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerID="69133d3632bcafa59cb764647713a7fc1a968abab529d60eb73a7d01ae35e503" exitCode=1
Sep 19 09:15:11.049850 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:11.049539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerDied","Data":"69133d3632bcafa59cb764647713a7fc1a968abab529d60eb73a7d01ae35e503"}
Sep 19 09:15:12.057218 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:12.057192    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/0.log"
Sep 19 09:15:12.057720 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:12.057262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerStarted","Data":"c8715534288108d45b5e5f9b59795725b9b0d9a9ff3df92cb3edd2cdc478ffac"}
Sep 19 09:15:13.405586 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:13.405554    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:13.406057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:13.405763    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer" containerID="cri-o://c8715534288108d45b5e5f9b59795725b9b0d9a9ff3df92cb3edd2cdc478ffac" gracePeriod=30
Sep 19 09:15:13.429364 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:13.429335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:14.067324 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.067300    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/1.log"
Sep 19 09:15:14.067683 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.067669    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/0.log"
Sep 19 09:15:14.067737 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.067699    2566 generic.go:356] "Generic (PLEG): container finished" podID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerID="c8715534288108d45b5e5f9b59795725b9b0d9a9ff3df92cb3edd2cdc478ffac" exitCode=1
Sep 19 09:15:14.067802 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.067748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerDied","Data":"c8715534288108d45b5e5f9b59795725b9b0d9a9ff3df92cb3edd2cdc478ffac"}
Sep 19 09:15:14.067846 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.067817    2566 scope.go:117] "RemoveContainer" containerID="69133d3632bcafa59cb764647713a7fc1a968abab529d60eb73a7d01ae35e503"
Sep 19 09:15:14.146941 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.146920    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/1.log"
Sep 19 09:15:14.150626 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.150610    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:14.277671 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config\") pod \"c97f006c-afb7-4464-a6be-e003dd680fe6\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") "
Sep 19 09:15:14.277671 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277649    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") pod \"c97f006c-afb7-4464-a6be-e003dd680fe6\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") "
Sep 19 09:15:14.277914 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location\") pod \"c97f006c-afb7-4464-a6be-e003dd680fe6\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") "
Sep 19 09:15:14.277914 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf6c8\" (UniqueName: \"kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8\") pod \"c97f006c-afb7-4464-a6be-e003dd680fe6\" (UID: \"c97f006c-afb7-4464-a6be-e003dd680fe6\") "
Sep 19 09:15:14.278033 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c97f006c-afb7-4464-a6be-e003dd680fe6" (UID: "c97f006c-afb7-4464-a6be-e003dd680fe6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:15:14.278033 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.277985    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c97f006c-afb7-4464-a6be-e003dd680fe6" (UID: "c97f006c-afb7-4464-a6be-e003dd680fe6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:14.279748 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.279728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c97f006c-afb7-4464-a6be-e003dd680fe6" (UID: "c97f006c-afb7-4464-a6be-e003dd680fe6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:15:14.279855 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.279747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8" (OuterVolumeSpecName: "kube-api-access-mf6c8") pod "c97f006c-afb7-4464-a6be-e003dd680fe6" (UID: "c97f006c-afb7-4464-a6be-e003dd680fe6"). InnerVolumeSpecName "kube-api-access-mf6c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:15:14.379204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.379170    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf6c8\" (UniqueName: \"kubernetes.io/projected/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-api-access-mf6c8\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:15:14.379204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.379197    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c97f006c-afb7-4464-a6be-e003dd680fe6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:15:14.379204 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.379207    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c97f006c-afb7-4464-a6be-e003dd680fe6-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:15:14.379435 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.379216    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c97f006c-afb7-4464-a6be-e003dd680fe6-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:15:14.545680 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.545551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:14.897447 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:14.897410    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:15.007826 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.007785    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:15.077816 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.077787    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd_c97f006c-afb7-4464-a6be-e003dd680fe6/storage-initializer/1.log"
Sep 19 09:15:15.078012 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.077880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd" event={"ID":"c97f006c-afb7-4464-a6be-e003dd680fe6","Type":"ContainerDied","Data":"0bc3c67d2aa144f2eff53960489709cffda3db99a017861587d145f0fece5eee"}
Sep 19 09:15:15.078012 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.077893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"
Sep 19 09:15:15.078012 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.077978    2566 scope.go:117] "RemoveContainer" containerID="c8715534288108d45b5e5f9b59795725b9b0d9a9ff3df92cb3edd2cdc478ffac"
Sep 19 09:15:15.126588 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.126558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:15.130846 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:15.130820    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-4snhd"]
Sep 19 09:15:16.180653 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:16.180619    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" path="/var/lib/kubelet/pods/c97f006c-afb7-4464-a6be-e003dd680fe6/volumes"
Sep 19 09:15:25.007696 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:25.007653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:31.560603 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:31.560560    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:35.007757 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:35.007719    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:45.007762 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:45.007722    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:15:51.290460 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:51.290427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:15:55.008146 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:15:55.008102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:16:05.007726 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:05.007636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:16:15.007353 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:15.007313    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:16:25.008411 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:25.008382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:16:32.796067 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:32.796033    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:16:32.796534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:32.796282    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" containerID="cri-o://b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c" gracePeriod=30
Sep 19 09:16:32.796534 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:32.796323    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kube-rbac-proxy" containerID="cri-o://baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3" gracePeriod=30
Sep 19 09:16:32.818213 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:32.818171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:16:33.360906 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:33.360875    2566 generic.go:356] "Generic (PLEG): container finished" podID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerID="baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3" exitCode=2
Sep 19 09:16:33.361073 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:33.360936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerDied","Data":"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"}
Sep 19 09:16:35.003451 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:35.003411    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 19 09:16:35.007688 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:35.007664    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 19 09:16:36.932555 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.932527    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:16:36.964576 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964546    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert\") pod \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") "
Sep 19 09:16:36.964723 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964593    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jsgp\" (UniqueName: \"kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp\") pod \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") "
Sep 19 09:16:36.964803 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location\") pod \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") "
Sep 19 09:16:36.964803 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config\") pod \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") "
Sep 19 09:16:36.964908 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") pod \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\" (UID: \"db10bd12-6e1f-4242-8c5b-a4a16488c0e4\") "
Sep 19 09:16:36.965027 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.964995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db10bd12-6e1f-4242-8c5b-a4a16488c0e4" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:16:36.965138 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.965001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db10bd12-6e1f-4242-8c5b-a4a16488c0e4" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:16:36.965138 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.965087    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:16:36.965226 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.965168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db10bd12-6e1f-4242-8c5b-a4a16488c0e4" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:16:36.966650 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.966627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp" (OuterVolumeSpecName: "kube-api-access-7jsgp") pod "db10bd12-6e1f-4242-8c5b-a4a16488c0e4" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4"). InnerVolumeSpecName "kube-api-access-7jsgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:16:36.966758 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:36.966743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db10bd12-6e1f-4242-8c5b-a4a16488c0e4" (UID: "db10bd12-6e1f-4242-8c5b-a4a16488c0e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:16:37.065568 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.065494    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsgp\" (UniqueName: \"kubernetes.io/projected/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-api-access-7jsgp\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:16:37.065568 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.065518    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:16:37.065568 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.065527    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:16:37.065568 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.065536    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db10bd12-6e1f-4242-8c5b-a4a16488c0e4-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:16:37.378155 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.378122    2566 generic.go:356] "Generic (PLEG): container finished" podID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerID="b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c" exitCode=0
Sep 19 09:16:37.378335 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.378166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerDied","Data":"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"}
Sep 19 09:16:37.378335 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.378179    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"
Sep 19 09:16:37.378335 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.378213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz" event={"ID":"db10bd12-6e1f-4242-8c5b-a4a16488c0e4","Type":"ContainerDied","Data":"fa4048a51ffe36b08739afcfef5ac23111c05c85b311e0e85408639aeaf8213c"}
Sep 19 09:16:37.378335 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.378239    2566 scope.go:117] "RemoveContainer" containerID="baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"
Sep 19 09:16:37.386420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.386399    2566 scope.go:117] "RemoveContainer" containerID="b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"
Sep 19 09:16:37.393254 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.393239    2566 scope.go:117] "RemoveContainer" containerID="50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"
Sep 19 09:16:37.399498 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.399474    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:16:37.400137 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400120    2566 scope.go:117] "RemoveContainer" containerID="baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"
Sep 19 09:16:37.400393 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:16:37.400376    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3\": container with ID starting with baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3 not found: ID does not exist" containerID="baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"
Sep 19 09:16:37.400449 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400402    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3"} err="failed to get container status \"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3\": rpc error: code = NotFound desc = could not find container \"baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3\": container with ID starting with baa78660cc4d81151b4e2461c1e21574d1e6b2d38ad17100bf4a8783b3ab5df3 not found: ID does not exist"
Sep 19 09:16:37.400449 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400420    2566 scope.go:117] "RemoveContainer" containerID="b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"
Sep 19 09:16:37.400639 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:16:37.400624    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c\": container with ID starting with b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c not found: ID does not exist" containerID="b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"
Sep 19 09:16:37.400682 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400645    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c"} err="failed to get container status \"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c\": rpc error: code = NotFound desc = could not find container \"b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c\": container with ID starting with b5e56c6fcebcb3e560b4db7884839411825923d9bbf6ade022e0dfc5bc124d5c not found: ID does not exist"
Sep 19 09:16:37.400682 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400658    2566 scope.go:117] "RemoveContainer" containerID="50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"
Sep 19 09:16:37.400879 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:16:37.400862    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd\": container with ID starting with 50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd not found: ID does not exist" containerID="50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"
Sep 19 09:16:37.400922 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.400884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd"} err="failed to get container status \"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd\": rpc error: code = NotFound desc = could not find container \"50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd\": container with ID starting with 50f3baae739f3d559d6afcd5c638d57764fab01590c6d028fb6eb8123d9b25cd not found: ID does not exist"
Sep 19 09:16:37.404271 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.404252    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xvvxz"]
Sep 19 09:16:37.959102 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.959065    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:16:37.989657 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989630    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:16:37.989900 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989887    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989901    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989922    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989933    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989941    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989946    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="storage-initializer"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989957    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kube-rbac-proxy"
Sep 19 09:16:37.989962 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.989963    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kube-rbac-proxy"
Sep 19 09:16:37.990275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.990004    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kserve-container"
Sep 19 09:16:37.990275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.990011    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.990275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.990017    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" containerName="kube-rbac-proxy"
Sep 19 09:16:37.990275 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.990022    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97f006c-afb7-4464-a6be-e003dd680fe6" containerName="storage-initializer"
Sep 19 09:16:37.994278 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.994263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:37.998142 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.998124    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 19 09:16:37.998142 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.998138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:16:37.998621 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.998603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:16:37.998821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.998805    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:16:37.998894 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.998805    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 19 09:16:37.999087 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.999072    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:16:37.999157 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:37.999111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:16:38.005658 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.005640    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:16:38.073038 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.073013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8lm\" (UniqueName: \"kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.073134 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.073046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.073134 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.073082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.073252 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.073144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.073252 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.073213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.173936 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.173911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174043 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.173950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8lm\" (UniqueName: \"kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174043 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.173969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174163 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.174080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174163 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.174130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174325 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.174306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174716 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.174695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.174754 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.174700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.176506 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.176480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.177538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.177515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:16:38.181045 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.181021    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db10bd12-6e1f-4242-8c5b-a4a16488c0e4" path="/var/lib/kubelet/pods/db10bd12-6e1f-4242-8c5b-a4a16488c0e4/volumes"
Sep 19 09:16:38.182211 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.182189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8lm\" (UniqueName: \"kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.304176 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.304123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:38.576824 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:38.576801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:16:38.578522 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:16:38.578496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52a87cc_3d00_4417_9fc4_b3741b5cda05.slice/crio-d10fca7fe3433c677076888efc23d7018db868a4b6a30b64a0e05e67beaac5e9 WatchSource:0}: Error finding container d10fca7fe3433c677076888efc23d7018db868a4b6a30b64a0e05e67beaac5e9: Status 404 returned error can't find the container with id d10fca7fe3433c677076888efc23d7018db868a4b6a30b64a0e05e67beaac5e9
Sep 19 09:16:39.388726 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:39.388686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerStarted","Data":"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"}
Sep 19 09:16:39.389212 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:39.388752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerStarted","Data":"d10fca7fe3433c677076888efc23d7018db868a4b6a30b64a0e05e67beaac5e9"}
Sep 19 09:16:42.401744 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:42.401710    2566 generic.go:356] "Generic (PLEG): container finished" podID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerID="bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7" exitCode=0
Sep 19 09:16:42.402083 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:42.401790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerDied","Data":"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"}
Sep 19 09:16:43.407355 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:43.407326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerStarted","Data":"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"}
Sep 19 09:16:43.407702 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:43.407383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerStarted","Data":"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"}
Sep 19 09:16:43.407702 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:43.407653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:43.428542 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:43.428502    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podStartSLOduration=2.802155001 podStartE2EDuration="6.428489468s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626334467s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:16:43.426689484 +0000 UTC m=+3367.747361068" watchObservedRunningTime="2026-09-19 09:16:43.428489468 +0000 UTC m=+3367.749161054"
Sep 19 09:16:44.416434 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:44.413821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:44.416827 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:44.416460    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:16:45.416020 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:45.415979    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:16:50.420723 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:50.420694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:16:50.421339 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:16:50.421311    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:00.422166 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:00.422123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:09.714363 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.714331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:17:09.748542 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.748510    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:09.752353 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.752338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:09.754685 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.754660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-8c0193\""
Sep 19 09:17:09.754811 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.754756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-8c0193-predictor-serving-cert\""
Sep 19 09:17:09.754811 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.754800    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-8c0193-kube-rbac-proxy-sar-config\""
Sep 19 09:17:09.755565 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.755551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-8c0193-dockercfg-6q96c\""
Sep 19 09:17:09.762221 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.762200    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:09.815260 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.815229    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:17:09.921382 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.921355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:09.921493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.921394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:09.921493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.921416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:09.921493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.921455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:09.921493 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:09.921479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022322 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022462 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022462 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022462 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022462 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.022826 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.022807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.023107 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.023086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.023176 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.023130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.024565 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.024548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.031205 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.031176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6\") pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") " pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.061900 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.061879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:10.332144 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.332121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:10.334161 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:17:10.334135    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7748859_6941_43c8_adda_445bf157c665.slice/crio-21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde WatchSource:0}: Error finding container 21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde: Status 404 returned error can't find the container with id 21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde
Sep 19 09:17:10.421458 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.421425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:10.506541 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.506508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerStarted","Data":"cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5"}
Sep 19 09:17:10.506679 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:10.506576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerStarted","Data":"21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde"}
Sep 19 09:17:16.531645 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:16.531622    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/0.log"
Sep 19 09:17:16.531945 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:16.531656    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7748859-6941-43c8-adda-445bf157c665" containerID="cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5" exitCode=1
Sep 19 09:17:16.531945 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:16.531693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerDied","Data":"cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5"}
Sep 19 09:17:17.537788 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:17.537745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/0.log"
Sep 19 09:17:17.538159 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:17.537868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerStarted","Data":"8fe58fa49db6ef2113a4a6cde4ed981a267d28964effd701017becdd935452cb"}
Sep 19 09:17:18.543630 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.543599    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/1.log"
Sep 19 09:17:18.544126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.544020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/0.log"
Sep 19 09:17:18.544126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.544049    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7748859-6941-43c8-adda-445bf157c665" containerID="8fe58fa49db6ef2113a4a6cde4ed981a267d28964effd701017becdd935452cb" exitCode=1
Sep 19 09:17:18.544126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.544079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerDied","Data":"8fe58fa49db6ef2113a4a6cde4ed981a267d28964effd701017becdd935452cb"}
Sep 19 09:17:18.544126 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.544129    2566 scope.go:117] "RemoveContainer" containerID="cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5"
Sep 19 09:17:18.544440 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:18.544423    2566 scope.go:117] "RemoveContainer" containerID="cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5"
Sep 19 09:17:18.556990 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:17:18.556960    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_kserve-ci-e2e-test_b7748859-6941-43c8-adda-445bf157c665_0 in pod sandbox 21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde from index: no such id: 'cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5'" containerID="cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5"
Sep 19 09:17:18.557071 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:17:18.557005    2566 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_kserve-ci-e2e-test_b7748859-6941-43c8-adda-445bf157c665_0 in pod sandbox 21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde from index: no such id: 'cb4661996dfe9d43eda8555fbe31c122f91c713ae0022c35a47c93c99ecfc5d5'; Skipping pod \"isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_kserve-ci-e2e-test(b7748859-6941-43c8-adda-445bf157c665)\"" logger="UnhandledError"
Sep 19 09:17:18.557210 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:17:18.557190    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_kserve-ci-e2e-test(b7748859-6941-43c8-adda-445bf157c665)\"" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" podUID="b7748859-6941-43c8-adda-445bf157c665"
Sep 19 09:17:19.549722 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:19.549693    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/1.log"
Sep 19 09:17:20.422009 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:20.421976    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:27.792391 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.792354    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:27.824243 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.824208    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:17:27.894646 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.894613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:17:27.907811 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.907792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/1.log"
Sep 19 09:17:27.909958 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.909943    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:27.957453 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls\") pod \"b7748859-6941-43c8-adda-445bf157c665\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") "
Sep 19 09:17:27.957626 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config\") pod \"b7748859-6941-43c8-adda-445bf157c665\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") "
Sep 19 09:17:27.957626 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6\") pod \"b7748859-6941-43c8-adda-445bf157c665\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") "
Sep 19 09:17:27.957626 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957542    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location\") pod \"b7748859-6941-43c8-adda-445bf157c665\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") "
Sep 19 09:17:27.957626 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert\") pod \"b7748859-6941-43c8-adda-445bf157c665\" (UID: \"b7748859-6941-43c8-adda-445bf157c665\") "
Sep 19 09:17:27.957900 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7748859-6941-43c8-adda-445bf157c665" (UID: "b7748859-6941-43c8-adda-445bf157c665"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:27.958011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.957950    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7748859-6941-43c8-adda-445bf157c665" (UID: "b7748859-6941-43c8-adda-445bf157c665"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:17:27.958011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.958001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7748859-6941-43c8-adda-445bf157c665" (UID: "b7748859-6941-43c8-adda-445bf157c665"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:17:27.959583 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.959562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7748859-6941-43c8-adda-445bf157c665" (UID: "b7748859-6941-43c8-adda-445bf157c665"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:17:27.959667 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:27.959645    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6" (OuterVolumeSpecName: "kube-api-access-wzjq6") pod "b7748859-6941-43c8-adda-445bf157c665" (UID: "b7748859-6941-43c8-adda-445bf157c665"). InnerVolumeSpecName "kube-api-access-wzjq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:17:28.058611 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.058523    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7748859-6941-43c8-adda-445bf157c665-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:17:28.058611 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.058557    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:17:28.058611 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.058567    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/b7748859-6941-43c8-adda-445bf157c665-kube-api-access-wzjq6\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:17:28.058611 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.058576    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7748859-6941-43c8-adda-445bf157c665-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:17:28.058611 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.058584    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7748859-6941-43c8-adda-445bf157c665-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:17:28.584015 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.583986    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg_b7748859-6941-43c8-adda-445bf157c665/storage-initializer/1.log"
Sep 19 09:17:28.584189 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.584068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg" event={"ID":"b7748859-6941-43c8-adda-445bf157c665","Type":"ContainerDied","Data":"21c76f1a35c4ad811ec1a301b355bfde85bc64959f6c09d37a2fc35592c84cde"}
Sep 19 09:17:28.584189 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.584086    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"
Sep 19 09:17:28.584189 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.584123    2566 scope.go:117] "RemoveContainer" containerID="8fe58fa49db6ef2113a4a6cde4ed981a267d28964effd701017becdd935452cb"
Sep 19 09:17:28.657719 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.657684    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:28.663323 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:28.663289    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-8c0193-predictor-6ccf65ff56-wdqpg"]
Sep 19 09:17:30.180413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:30.180381    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7748859-6941-43c8-adda-445bf157c665" path="/var/lib/kubelet/pods/b7748859-6941-43c8-adda-445bf157c665/volumes"
Sep 19 09:17:30.421850 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:30.421803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:40.421643 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:40.421602    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:50.422034 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:50.421988    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:17:56.178079 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:17:56.178033    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 19 09:18:06.179933 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:06.179905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:18:09.575316 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.575279    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:18:09.575782 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.575539    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container" containerID="cri-o://12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286" gracePeriod=30
Sep 19 09:18:09.575782 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.575612    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kube-rbac-proxy" containerID="cri-o://e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125" gracePeriod=30
Sep 19 09:18:09.599222 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.599194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:09.748705 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.748676    2566 generic.go:356] "Generic (PLEG): container finished" podID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerID="e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125" exitCode=2
Sep 19 09:18:09.748848 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:09.748747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerDied","Data":"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"}
Sep 19 09:18:10.417045 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:10.417008    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 19 09:18:13.513066 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.513038    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:18:13.603207 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config\") pod \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") "
Sep 19 09:18:13.603207 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location\") pod \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") "
Sep 19 09:18:13.603207 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603198    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert\") pod \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") "
Sep 19 09:18:13.603413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8lm\" (UniqueName: \"kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm\") pod \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") "
Sep 19 09:18:13.603413 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603261    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls\") pod \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\" (UID: \"a52a87cc-3d00-4417-9fc4-b3741b5cda05\") "
Sep 19 09:18:13.603525 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603463    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a52a87cc-3d00-4417-9fc4-b3741b5cda05" (UID: "a52a87cc-3d00-4417-9fc4-b3741b5cda05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:13.603581 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603538    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a52a87cc-3d00-4417-9fc4-b3741b5cda05" (UID: "a52a87cc-3d00-4417-9fc4-b3741b5cda05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:18:13.603581 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.603549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a52a87cc-3d00-4417-9fc4-b3741b5cda05" (UID: "a52a87cc-3d00-4417-9fc4-b3741b5cda05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:18:13.605240 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.605219    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a52a87cc-3d00-4417-9fc4-b3741b5cda05" (UID: "a52a87cc-3d00-4417-9fc4-b3741b5cda05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:18:13.605310 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.605251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm" (OuterVolumeSpecName: "kube-api-access-lt8lm") pod "a52a87cc-3d00-4417-9fc4-b3741b5cda05" (UID: "a52a87cc-3d00-4417-9fc4-b3741b5cda05"). InnerVolumeSpecName "kube-api-access-lt8lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:18:13.704545 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.704515    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:13.704545 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.704542    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:13.704693 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.704552    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a52a87cc-3d00-4417-9fc4-b3741b5cda05-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:13.704693 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.704561    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8lm\" (UniqueName: \"kubernetes.io/projected/a52a87cc-3d00-4417-9fc4-b3741b5cda05-kube-api-access-lt8lm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:13.704693 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.704569    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52a87cc-3d00-4417-9fc4-b3741b5cda05-proxy-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:13.764430 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.764404    2566 generic.go:356] "Generic (PLEG): container finished" podID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerID="12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286" exitCode=0
Sep 19 09:18:13.764530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.764437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerDied","Data":"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"}
Sep 19 09:18:13.764530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.764455    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"
Sep 19 09:18:13.764530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.764490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k" event={"ID":"a52a87cc-3d00-4417-9fc4-b3741b5cda05","Type":"ContainerDied","Data":"d10fca7fe3433c677076888efc23d7018db868a4b6a30b64a0e05e67beaac5e9"}
Sep 19 09:18:13.764530 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.764517    2566 scope.go:117] "RemoveContainer" containerID="e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"
Sep 19 09:18:13.772235 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.772216    2566 scope.go:117] "RemoveContainer" containerID="12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"
Sep 19 09:18:13.779471 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.779454    2566 scope.go:117] "RemoveContainer" containerID="bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"
Sep 19 09:18:13.785607 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.785587    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:18:13.786318 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.786306    2566 scope.go:117] "RemoveContainer" containerID="e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"
Sep 19 09:18:13.786579 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:13.786561    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125\": container with ID starting with e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125 not found: ID does not exist" containerID="e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"
Sep 19 09:18:13.786642 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.786604    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125"} err="failed to get container status \"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125\": rpc error: code = NotFound desc = could not find container \"e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125\": container with ID starting with e4ee13ad9e4a99541031751c283c0dcc8b792e34e1dc1969280f26f414fbb125 not found: ID does not exist"
Sep 19 09:18:13.786642 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.786629    2566 scope.go:117] "RemoveContainer" containerID="12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"
Sep 19 09:18:13.786944 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:13.786927    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286\": container with ID starting with 12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286 not found: ID does not exist" containerID="12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"
Sep 19 09:18:13.786988 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.786949    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286"} err="failed to get container status \"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286\": rpc error: code = NotFound desc = could not find container \"12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286\": container with ID starting with 12795a445c3c25ccf89c3b31bf4d9e038ed5496d5eb9130aaf9a3b12a7092286 not found: ID does not exist"
Sep 19 09:18:13.786988 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.786962    2566 scope.go:117] "RemoveContainer" containerID="bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"
Sep 19 09:18:13.787197 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:13.787181    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7\": container with ID starting with bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7 not found: ID does not exist" containerID="bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"
Sep 19 09:18:13.787239 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.787204    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7"} err="failed to get container status \"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7\": rpc error: code = NotFound desc = could not find container \"bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7\": container with ID starting with bd0f1ca4ad61701249b624b9ce1a3a916eadba82dd2f2f87a22df6056a7460d7 not found: ID does not exist"
Sep 19 09:18:13.790176 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:13.790158    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5cd2k"]
Sep 19 09:18:14.180426 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:14.180398    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" path="/var/lib/kubelet/pods/a52a87cc-3d00-4417-9fc4-b3741b5cda05/volumes"
Sep 19 09:18:16.483823 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.483785    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8pqsf/must-gather-cwdsb"]
Sep 19 09:18:16.484483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484456    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container"
Sep 19 09:18:16.484483 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484485    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484506    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="storage-initializer"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484513    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="storage-initializer"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484525    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484533    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484559    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kube-rbac-proxy"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484567    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kube-rbac-proxy"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484628    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kserve-container"
Sep 19 09:18:16.484640 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484640    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.485053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484650    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a52a87cc-3d00-4417-9fc4-b3741b5cda05" containerName="kube-rbac-proxy"
Sep 19 09:18:16.485053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484661    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.485053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484739    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.485053 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.484803    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7748859-6941-43c8-adda-445bf157c665" containerName="storage-initializer"
Sep 19 09:18:16.489260 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.489239    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.491732 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.491712    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8pqsf\"/\"kube-root-ca.crt\""
Sep 19 09:18:16.491857 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.491716    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8pqsf\"/\"openshift-service-ca.crt\""
Sep 19 09:18:16.496025 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.495994    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8pqsf/must-gather-cwdsb"]
Sep 19 09:18:16.505359 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.505334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:16.585473 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.585439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:16.622552 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.622528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.622643 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.622567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rh7t\" (UniqueName: \"kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.687021 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.686988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:16.723443 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.723422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.723505 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.723458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rh7t\" (UniqueName: \"kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.723748 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.723730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.731936 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.731913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rh7t\" (UniqueName: \"kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t\") pod \"must-gather-cwdsb\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") " pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:16.798266 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:16.798216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:17.071506 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:17.071427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8pqsf/must-gather-cwdsb"]
Sep 19 09:18:17.074052 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:18:17.074026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8a37ca_3d52_470d_92d7_74d004dd73a7.slice/crio-7e8aff19c04206b237802755b4e6834171a91874800f2b804f84200c52fa6ced WatchSource:0}: Error finding container 7e8aff19c04206b237802755b4e6834171a91874800f2b804f84200c52fa6ced: Status 404 returned error can't find the container with id 7e8aff19c04206b237802755b4e6834171a91874800f2b804f84200c52fa6ced
Sep 19 09:18:17.075815 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:17.075794    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:18:17.784738 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:17.784688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" event={"ID":"0f8a37ca-3d52-470d-92d7-74d004dd73a7","Type":"ContainerStarted","Data":"7e8aff19c04206b237802755b4e6834171a91874800f2b804f84200c52fa6ced"}
Sep 19 09:18:23.810398 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:23.810363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" event={"ID":"0f8a37ca-3d52-470d-92d7-74d004dd73a7","Type":"ContainerStarted","Data":"f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51"}
Sep 19 09:18:23.810873 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:23.810411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" event={"ID":"0f8a37ca-3d52-470d-92d7-74d004dd73a7","Type":"ContainerStarted","Data":"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"}
Sep 19 09:18:23.827907 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:23.827861    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" podStartSLOduration=1.9500310079999998 podStartE2EDuration="7.827845099s" totalImagesPullingTime="5.877814091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:18:23.827800569 +0000 UTC m=+3468.148472156" watchObservedRunningTime="2026-09-19 09:18:23.827845099 +0000 UTC m=+3468.148516677"
Sep 19 09:18:49.910124 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:49.910092    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerID="c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a" exitCode=0
Sep 19 09:18:49.910508 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:49.910164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" event={"ID":"0f8a37ca-3d52-470d-92d7-74d004dd73a7","Type":"ContainerDied","Data":"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"}
Sep 19 09:18:49.910508 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:49.910450    2566 scope.go:117] "RemoveContainer" containerID="c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"
Sep 19 09:18:50.558343 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:50.558318    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8pqsf_must-gather-cwdsb_0f8a37ca-3d52-470d-92d7-74d004dd73a7/gather/0.log"
Sep 19 09:18:51.107897 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:51.107866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:51.263670 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:51.263641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:51.364235 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:51.364203    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:54.133040 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:54.133003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h7m8x_16c58e2d-7fe3-436e-93d1-68bd2ed8385b/global-pull-secret-syncer/0.log"
Sep 19 09:18:54.305826 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:54.305797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log"
Sep 19 09:18:54.412471 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:54.412392    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-228.ec2.internal_2692f115a698526b3ec4e4ebc7f1819f/haproxy/0.log"
Sep 19 09:18:56.095358 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.095307    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8pqsf/must-gather-cwdsb"]
Sep 19 09:18:56.095795 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.095508    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-8pqsf/must-gather-cwdsb" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="copy" containerID="cri-o://f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51" gracePeriod=2
Sep 19 09:18:56.100140 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.100112    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8pqsf/must-gather-cwdsb"]
Sep 19 09:18:56.179601 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:56.179563    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cwdsb\" is forbidden: User \"system:node:ip-10-0-136-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8pqsf\": no relationship found between node 'ip-10-0-136-228.ec2.internal' and this object" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:56.322405 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.322383    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8pqsf_must-gather-cwdsb_0f8a37ca-3d52-470d-92d7-74d004dd73a7/copy/0.log"
Sep 19 09:18:56.325000 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.324984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:56.454120 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.454087    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rh7t\" (UniqueName: \"kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t\") pod \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") "
Sep 19 09:18:56.454280 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.454148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output\") pod \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\" (UID: \"0f8a37ca-3d52-470d-92d7-74d004dd73a7\") "
Sep 19 09:18:56.455576 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.455543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0f8a37ca-3d52-470d-92d7-74d004dd73a7" (UID: "0f8a37ca-3d52-470d-92d7-74d004dd73a7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:56.456263 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.456248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t" (OuterVolumeSpecName: "kube-api-access-6rh7t") pod "0f8a37ca-3d52-470d-92d7-74d004dd73a7" (UID: "0f8a37ca-3d52-470d-92d7-74d004dd73a7"). InnerVolumeSpecName "kube-api-access-6rh7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:18:56.555409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.555383    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rh7t\" (UniqueName: \"kubernetes.io/projected/0f8a37ca-3d52-470d-92d7-74d004dd73a7-kube-api-access-6rh7t\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:56.555409 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.555404    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8a37ca-3d52-470d-92d7-74d004dd73a7-must-gather-output\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 19 09:18:56.828633 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.828550    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:56.929680 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.929638    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56xxt_bf05c100-c217-44d7-a410-b5e695f03388/konnectivity-agent/0.log" limit=1024
Sep 19 09:18:56.937191 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.937172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8pqsf_must-gather-cwdsb_0f8a37ca-3d52-470d-92d7-74d004dd73a7/copy/0.log"
Sep 19 09:18:56.937519 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.937499    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerID="f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51" exitCode=143
Sep 19 09:18:56.937628 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.937542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8pqsf/must-gather-cwdsb"
Sep 19 09:18:56.937628 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.937555    2566 scope.go:117] "RemoveContainer" containerID="f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51"
Sep 19 09:18:56.945336 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.945231    2566 scope.go:117] "RemoveContainer" containerID="c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"
Sep 19 09:18:56.957739 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.957721    2566 scope.go:117] "RemoveContainer" containerID="f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51"
Sep 19 09:18:56.958053 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:56.958034    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51\": container with ID starting with f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51 not found: ID does not exist" containerID="f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51"
Sep 19 09:18:56.958108 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.958063    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51"} err="failed to get container status \"f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51\": rpc error: code = NotFound desc = could not find container \"f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51\": container with ID starting with f32374a2d214ac4b981d72f347707e2c1296340ebcc3eeb82c2ccba1076b9a51 not found: ID does not exist"
Sep 19 09:18:56.958108 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.958083    2566 scope.go:117] "RemoveContainer" containerID="c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"
Sep 19 09:18:56.958343 ip-10-0-136-228 kubenswrapper[2566]: E0919 09:18:56.958325    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a\": container with ID starting with c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a not found: ID does not exist" containerID="c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"
Sep 19 09:18:56.958382 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:56.958349    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a"} err="failed to get container status \"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a\": rpc error: code = NotFound desc = could not find container \"c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a\": container with ID starting with c528ebe5c55d57f432ae22dd2e01c38f8f7f5fbdc3a32f2a0390c55dc1c0614a not found: ID does not exist"
Sep 19 09:18:58.180810 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:58.180714    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" path="/var/lib/kubelet/pods/0f8a37ca-3d52-470d-92d7-74d004dd73a7/volumes"
Sep 19 09:18:59.036628 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.036545    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzj4q_ff7543c6-457a-43c4-8beb-32e8a9e76d9f/node-exporter/0.log"
Sep 19 09:18:59.057272 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.057242    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzj4q_ff7543c6-457a-43c4-8beb-32e8a9e76d9f/kube-rbac-proxy/0.log"
Sep 19 09:18:59.078179 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.078156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzj4q_ff7543c6-457a-43c4-8beb-32e8a9e76d9f/init-textfile/0.log"
Sep 19 09:18:59.208471 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.208444    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/prometheus/0.log"
Sep 19 09:18:59.228137 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.228109    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/config-reloader/0.log"
Sep 19 09:18:59.249832 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.249806    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/thanos-sidecar/0.log"
Sep 19 09:18:59.274804 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.274781    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/kube-rbac-proxy-web/0.log"
Sep 19 09:18:59.301510 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.301431    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/kube-rbac-proxy/0.log"
Sep 19 09:18:59.324878 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.324853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/kube-rbac-proxy-thanos/0.log"
Sep 19 09:18:59.348080 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.348052    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_08b6efbc-53c8-4cd9-8336-958777ba81b1/init-config-reloader/0.log"
Sep 19 09:18:59.390864 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.390831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c2jqk_d60b80c4-7785-49b5-9c57-6539dbcc0d95/prometheus-operator/0.log"
Sep 19 09:18:59.408011 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:18:59.407987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c2jqk_d60b80c4-7785-49b5-9c57-6539dbcc0d95/kube-rbac-proxy/0.log"
Sep 19 09:19:02.933764 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.933728    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"]
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934010    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="gather"
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934023    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="gather"
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="copy"
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934037    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="copy"
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934080    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="copy"
Sep 19 09:19:02.934145 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.934089    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8a37ca-3d52-470d-92d7-74d004dd73a7" containerName="gather"
Sep 19 09:19:02.939139 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.939117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:02.941669 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.941639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-582w2\"/\"default-dockercfg-phx2h\""
Sep 19 09:19:02.941669 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.941665    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-582w2\"/\"openshift-service-ca.crt\""
Sep 19 09:19:02.942641 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.942622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-582w2\"/\"kube-root-ca.crt\""
Sep 19 09:19:02.947484 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:02.947464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"]
Sep 19 09:19:03.005659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.005619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-lib-modules\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.005659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.005660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knxgs\" (UniqueName: \"kubernetes.io/projected/d8b4b51a-853a-4633-b4dd-c92177c4e590-kube-api-access-knxgs\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.005909 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.005684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-podres\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.005909 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.005785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-proc\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.005909 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.005821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-sys\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107014 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.106973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-proc\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107014 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-sys\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-lib-modules\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxgs\" (UniqueName: \"kubernetes.io/projected/d8b4b51a-853a-4633-b4dd-c92177c4e590-kube-api-access-knxgs\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-proc\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-sys\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-podres\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107269 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-lib-modules\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.107538 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.107308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b4b51a-853a-4633-b4dd-c92177c4e590-podres\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.115143 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.115114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxgs\" (UniqueName: \"kubernetes.io/projected/d8b4b51a-853a-4633-b4dd-c92177c4e590-kube-api-access-knxgs\") pod \"perf-node-gather-daemonset-hzqvq\" (UID: \"d8b4b51a-853a-4633-b4dd-c92177c4e590\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.250420 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.250324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.520412 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.520331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"]
Sep 19 09:19:03.523155 ip-10-0-136-228 kubenswrapper[2566]: W0919 09:19:03.523130    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8b4b51a_853a_4633_b4dd_c92177c4e590.slice/crio-9f2f3b6c4ecba849cea99f3708419d634ed40ec7c15c6e7d9deba9800b580f1b WatchSource:0}: Error finding container 9f2f3b6c4ecba849cea99f3708419d634ed40ec7c15c6e7d9deba9800b580f1b: Status 404 returned error can't find the container with id 9f2f3b6c4ecba849cea99f3708419d634ed40ec7c15c6e7d9deba9800b580f1b
Sep 19 09:19:03.643227 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.643197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfbdv_57e04927-ba9e-458f-bd24-47f4bacc2896/dns/0.log"
Sep 19 09:19:03.664408 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.664387    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfbdv_57e04927-ba9e-458f-bd24-47f4bacc2896/kube-rbac-proxy/0.log"
Sep 19 09:19:03.737005 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.736979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vf9n9_f1704028-5d63-44ab-b038-4867a45a8896/dns-node-resolver/0.log"
Sep 19 09:19:03.965399 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.965366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq" event={"ID":"d8b4b51a-853a-4633-b4dd-c92177c4e590","Type":"ContainerStarted","Data":"f0eeb2648ff45949c7984f2520238299f8ea2c3f3b36237acd31e42254ca8f00"}
Sep 19 09:19:03.965821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.965419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq" event={"ID":"d8b4b51a-853a-4633-b4dd-c92177c4e590","Type":"ContainerStarted","Data":"9f2f3b6c4ecba849cea99f3708419d634ed40ec7c15c6e7d9deba9800b580f1b"}
Sep 19 09:19:03.965821 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.965464    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:03.987804 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:03.987729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq" podStartSLOduration=1.98771048 podStartE2EDuration="1.98771048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:19:03.986384909 +0000 UTC m=+3508.307056498" watchObservedRunningTime="2026-09-19 09:19:03.98771048 +0000 UTC m=+3508.308382068"
Sep 19 09:19:04.239764 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:04.239675    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c5868599f-tqmnc_c6a8e307-6e55-42ed-b5d0-8b0067caf21a/registry/0.log"
Sep 19 09:19:04.266706 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:04.266669    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hdz7_e93a1df2-a189-4c33-aad1-d17ca2d7cc1d/node-ca/0.log"
Sep 19 09:19:05.442057 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:05.442024    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8drv4_d47e6619-183e-4631-a052-55895ec8508f/serve-healthcheck-canary/0.log"
Sep 19 09:19:05.850826 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:05.850734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gxjl_4c6f8397-5311-42a8-84b1-6416370ef901/kube-rbac-proxy/0.log"
Sep 19 09:19:05.871235 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:05.871206    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gxjl_4c6f8397-5311-42a8-84b1-6416370ef901/exporter/0.log"
Sep 19 09:19:05.893517 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:05.893487    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4gxjl_4c6f8397-5311-42a8-84b1-6416370ef901/extractor/0.log"
Sep 19 09:19:09.028882 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.028839    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vxtpn_a2864471-f847-4b29-9fa1-b02b4d4bb698/s3-init/0.log"
Sep 19 09:19:09.047340 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.047313    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vxtpn_a2864471-f847-4b29-9fa1-b02b4d4bb698/download-hf-model/0.log"
Sep 19 09:19:09.080923 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.080900    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sh67h_c2aedc62-185d-438d-8a1b-55c8243125a9/s3-tls-init-custom/0.log"
Sep 19 09:19:09.098077 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.098055    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sh67h_c2aedc62-185d-438d-8a1b-55c8243125a9/download-hf-model/0.log"
Sep 19 09:19:09.132735 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.132705    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f5wgw_4ddf3167-2288-428b-9949-feddb1309009/s3-tls-init-serving/0.log"
Sep 19 09:19:09.150201 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.150171    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f5wgw_4ddf3167-2288-428b-9949-feddb1309009/download-hf-model/0.log"
Sep 19 09:19:09.977485 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:09.977453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-hzqvq"
Sep 19 09:19:15.216824 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.216796    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-99ch6_2529d58c-8035-4da8-9411-00283e9e465d/kube-multus/0.log"
Sep 19 09:19:15.242121 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.242090    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/kube-multus-additional-cni-plugins/0.log"
Sep 19 09:19:15.264659 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.264638    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/egress-router-binary-copy/0.log"
Sep 19 09:19:15.284797 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.284759    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/cni-plugins/0.log"
Sep 19 09:19:15.336784 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.336738    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/bond-cni-plugin/0.log"
Sep 19 09:19:15.356618 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.356598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/routeoverride-cni/0.log"
Sep 19 09:19:15.376273 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.376235    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/whereabouts-cni-bincopy/0.log"
Sep 19 09:19:15.396554 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.396534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mlvx_7d2b547b-9039-4c47-851b-9c6d065e56d8/whereabouts-cni/0.log"
Sep 19 09:19:15.828391 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.828361    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c4vdw_f6df42b6-9d4c-4388-8750-2b7e19ccbe71/network-metrics-daemon/0.log"
Sep 19 09:19:15.849147 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:15.849121    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c4vdw_f6df42b6-9d4c-4388-8750-2b7e19ccbe71/kube-rbac-proxy/0.log"
Sep 19 09:19:16.824734 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.824703    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/ovn-controller/0.log"
Sep 19 09:19:16.842290 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.842263    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/ovn-acl-logging/0.log"
Sep 19 09:19:16.873554 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.873527    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/kube-rbac-proxy-node/0.log"
Sep 19 09:19:16.901104 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.901078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 09:19:16.918720 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.918688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/northd/0.log"
Sep 19 09:19:16.937537 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.937512    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/nbdb/0.log"
Sep 19 09:19:16.958807 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:16.958786    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/sbdb/0.log"
Sep 19 09:19:17.021207 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:17.021181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-clzwt_d861b7f9-efdf-4245-b57b-9ac9973cca19/ovnkube-controller/0.log"
Sep 19 09:19:18.275112 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:18.275082    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dtq79_1563caa6-c545-42d5-a4e0-11abe8191e82/network-check-target-container/0.log"
Sep 19 09:19:19.199245 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:19.199221    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-cm2b2_3921dcbc-5bec-4b64-a499-2a9340efaf3e/iptables-alerter/0.log"
Sep 19 09:19:19.898178 ip-10-0-136-228 kubenswrapper[2566]: I0919 09:19:19.898154    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vbccp_8ea52622-a3a6-482d-9a0e-dcb6df7a8bf4/tuned/0.log"