Sep 17 21:58:13.931849 ip-10-0-136-9 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:17.240611 ip-10-0-136-9 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:17.240611 ip-10-0-136-9 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:17.240611 ip-10-0-136-9 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:17.240611 ip-10-0-136-9 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:17.240611 ip-10-0-136-9 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:17.241507 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.241415    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:17.245134 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245114    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.245134 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245135    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245140    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245144    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245147    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245151    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245154    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245157    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245163    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245166    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245169    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245172    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245175    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245178    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245181    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245183    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245186    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245188    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245191    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245194    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245196    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.245209 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245199    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245202    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245205    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245207    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245211    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245214    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245217    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245220    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245224    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245226    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245230    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245234    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245237    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245239    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245242    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245245    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245247    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245250    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245252    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245255    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.245745 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245258    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245261    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245263    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245266    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245269    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245271    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245274    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245276    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245279    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245282    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245284    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245286    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245289    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245293    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245297    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245300    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245303    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245306    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245309    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245312    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.246236 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245314    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245317    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245319    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245322    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245325    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245327    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245330    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245333    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245335    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245338    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245340    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245343    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245345    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245350    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245352    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245356    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245360    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245363    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245366    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.246736 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245369    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245372    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245375    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245377    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245380    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245383    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245386    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245390    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245393    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245396    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245398    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245401    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245404    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245406    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245409    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245412    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245414    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245417    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.245419    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247017    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.247199 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247026    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247029    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247032    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247036    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247039    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247042    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247045    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247048    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247051    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247055    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247058    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247061    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247064    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247067    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247070    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247072    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247076    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247080    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247083    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.247728 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247086    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247089    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247092    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247095    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247098    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247100    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247104    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247106    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247109    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247112    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247129    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247132    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247136    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247139    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247142    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247145    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247148    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247151    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247155    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247158    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.248470 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247161    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247165    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247168    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247171    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247175    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247178    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247182    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247186    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247188    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247191    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247194    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247197    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247200    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247203    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247206    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247209    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247212    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247215    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.249069 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247218    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247220    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247223    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247225    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247228    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247231    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247233    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247236    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247239    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247242    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247245    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247247    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247250    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247252    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247255    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247257    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247260    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247263    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247266    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247269    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.249530 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247271    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247274    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247276    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247279    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247281    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247284    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247287    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247290    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247292    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247295    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247298    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247300    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247303    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247306    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247309    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247311    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247314    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247317    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247320    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247322    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.250070 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.247326    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248282    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248299    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248305    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248310    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248315    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248321    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248325    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248330    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248341    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248345    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248349    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248354    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248359    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248363    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248367    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248371    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248375    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248379    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248384    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.250581 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248388    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248392    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248402    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248406    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248411    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248415    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248419    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248424    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248428    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248432    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248436    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248440    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248445    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248449    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248459    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248468    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248474    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248478    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248483    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248487    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.251151 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248492    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248496    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248500    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248504    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248509    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248513    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248518    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248526    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248530    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248535    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248539    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248543    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248564    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248569    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248573    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248577    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248582    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248587    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248591    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.251656 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248611    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248640    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248650    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248656    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248661    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248666    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248674    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248678    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248681    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248684    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248689    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248711    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248718    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248722    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248727    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248853    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248862    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248867    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248874    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.252115 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248877    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248880    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248883    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248886    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248889    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248892    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248897    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248900    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248903    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248905    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248908    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248911    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248913    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248918    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248922    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248925    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248928    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248933    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248936    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.252630 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248939    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248942    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.248945    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249029    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249063    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249071    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249076    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249081    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249085    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249090    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249095    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249098    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249103    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249107    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249110    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249114    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249117    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249120    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249123    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249125    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249128    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249133    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249136    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:17.253091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249139    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249142    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249146    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249150    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249153    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249156    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249159    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249162    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249165    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249168    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249171    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249174    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249178    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249181    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249185    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249188    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249191    2582 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249193    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249199    2582 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249202    2582 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249205    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249210    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249213    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249217    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:17.253685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249220    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249223    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249226    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249229    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249232    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249234    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249238    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249241    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249244    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249247    2582 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249250    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249253    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249256    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249259    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249262    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249265    2582 flags.go:64] FLAG: --help="false"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249268    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249271    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249274    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249277    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249280    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249284    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249287    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:17.254271 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249290    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249293    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249296    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249298    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249301    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249304    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249309    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249312    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249315    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249318    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249321    2582 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249324    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249327    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249330    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249336    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249338    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249342    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249345    2582 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249348    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249351    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249354    2582 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249357    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249361    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249365    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249369    2582 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:17.254880 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249372    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249375    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249378    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249381    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249384    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249387    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249390    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249397    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249402    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249405    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249408    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249411    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249416    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249421    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249424    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249427    2582 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249430    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249433    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00b7a2fe1db57b90b"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249436    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249440    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249442    2582 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249445    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249449    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249452    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:17.255608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249455    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249458    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249462    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249465    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249468    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249471    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249474    2582 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249477    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249480    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249483    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249486    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249488    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249491    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249495    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249498    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249501    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249504    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249508    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249511    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249514    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249517    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249521    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249527    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249530    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249533    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:17.256707 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249537    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249540    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249543    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249562    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249565    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249569    2582 flags.go:64] FLAG: --v="2"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249573    2582 flags.go:64] FLAG: --version="false"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249577    2582 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249581    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.249585    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249717    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249721    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249725    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249728    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249731    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249735    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249738    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249741    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249744    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249747    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249749    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249752    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.257780 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249755    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249758    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249762    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249765    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249767    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249770    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249773    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249776    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249779    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249783    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249786    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249789    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249792    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249795    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249798    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249801    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249804    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249806    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249809    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249812    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.258784 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249815    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249818    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249820    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249823    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249825    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249828    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249831    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249833    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249836    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249838    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249841    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249844    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249847    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249849    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249853    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249856    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249859    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249862    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249864    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249867    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.259533 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249870    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249873    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249877    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249881    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249884    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249886    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249889    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249892    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249895    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249897    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249900    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249903    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249906    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249908    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249911    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249914    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249917    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249919    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249923    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.260251 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249926    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249929    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249932    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249935    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249937    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249940    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249942    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249947    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249949    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249952    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249955    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249958    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249961    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249964    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249967    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249970    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249972    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249975    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249977    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.261060 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249980    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249983    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249986    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249989    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249991    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249994    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.249997    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.250000    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.250003    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.250011    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.257730    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.257754    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.257783    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.257796    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257883    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.261699 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257891    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257896    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257901    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257906    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257910    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257915    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257919    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257923    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257928    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257932    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257936    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257941    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257945    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257950    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257955    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257959    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257964    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257969    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257973    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257977    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.262250 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257982    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257986    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257990    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257994    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.257999    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258003    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258007    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258011    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258018    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258023    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258027    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258032    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258036    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258040    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258045    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258049    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258054    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258058    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258062    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258066    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.262855 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258071    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258075    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258082    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258089    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258093    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258098    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258103    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258107    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258111    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258115    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258142    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258149    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258155    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258160    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258165    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258171    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258175    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258181    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258185    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.263388 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258189    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258195    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258201    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258206    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258210    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258215    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258219    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258224    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258228    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258233    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258238    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258242    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258246    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258251    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258255    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258260    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258264    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258268    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258272    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258276    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.263930 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258280    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258285    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258289    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258293    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258297    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258302    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258306    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258310    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258315    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258319    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258323    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258327    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258334    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258341    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258347    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258355    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258360    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258364    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.264436 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258369    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.258378    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258588    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258600    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258605    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258611    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258616    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258621    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258626    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258630    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258635    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258640    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258645    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258649    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:17.264896 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258653    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258658    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258662    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258668    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258673    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258678    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258682    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258687    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258691    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258695    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258700    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258704    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258709    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258714    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258719    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258723    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258728    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258733    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258737    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258744    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:17.265435 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258750    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258755    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258760    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258765    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258769    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258773    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258777    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258782    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258785    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258789    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258795    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258800    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258805    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258809    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258814    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258818    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258822    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258827    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258831    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:17.266255 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258835    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258839    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258844    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258848    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258852    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258856    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258860    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258865    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258869    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258875    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258881    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258886    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258890    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258895    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258900    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258904    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258908    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258913    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258917    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258922    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:17.267061 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258926    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258930    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258934    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258939    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258943    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258947    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258952    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258956    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258960    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258964    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258969    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258973    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258977    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258981    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258985    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258990    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258994    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.258998    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259002    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259007    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:17.267883 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259012    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259016    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259020    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259025    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259030    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259034    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259038    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:17.259042    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.259050    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.259226    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.261828    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.262843    2582 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.262951    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:17.268391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.262985    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:17.268745 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.265626    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:17.268745 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.267430    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:17.272151 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.272134    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:17.274132 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.274115    2582 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:17.275005 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.274991    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:17.277259 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.277233    2582 fs.go:135] Filesystem UUIDs: map[20d8cc87-17a2-4085-80fb-147a1483efb0:/dev/nvme0n1p4 2fff58b2-99c3-4697-87f0-552438d68dbe:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 21:58:17.277344 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.277259    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:17.283072 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.282944    2582 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:17.282408432 +0000 UTC m=+1.950471169 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103968 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5f04b390c85e519daa41bca534a4 SystemUUID:ec2b5f04-b390-c85e-519d-aa41bca534a4 BootID:72e02c26-3225-47b7-8864-d524b8b83961 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:6f:53:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:6f:53:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:f9:79:ed:d3:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:17.283072 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.283058    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:17.283260 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.283146    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:17.284817 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.284784    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:17.284957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.284820    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:17.285468 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.285458    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:17.285497 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.285472    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:17.285497 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.285481    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:17.285497 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.285493    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:17.286627 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.286616    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:17.290713 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.290698    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:17.291021 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291010    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:17.291062 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291048    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:17.291062 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291059    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:17.291125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291074    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:17.291125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291083    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:17.291125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291093    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:17.291455 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291440    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:17.291518 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.291463    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:17.298265 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.298241    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:17.300048 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.300028    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:17.301905 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.301887    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:17.301987 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.301927    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:17.302878 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302864    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302884    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302893    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302902    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302916    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302925    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302945    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:17.302951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302954    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302966    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302975    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302989    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.302998    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.303026    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:17.303194 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.303035    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:17.305468 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.305446    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:17.306128 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.306099    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:17.306205 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.306098    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:17.307058 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.307042    2582 server.go:1268] "Started kubelet"
Sep 17 21:58:17.307127 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.307098    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:17.307168 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.307120    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:17.307206 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.307191    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:17.307997 ip-10-0-136-9 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:17.308651 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.308232    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:17.308651 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.308355    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:17.312200 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.312180    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:17.314651 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.314635    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:17.314651 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.314650    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:17.314809 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.314771    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.315225 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315210    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:17.315309 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315212    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:17.315309 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315243    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:17.315309 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315296    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:17.315427 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315315    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:17.315703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315681    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:17.315703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315702    2582 factory.go:55] Registering systemd factory
Sep 17 21:58:17.315871 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.315723    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:17.316020 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.316008    2582 factory.go:153] Registering CRI-O factory
Sep 17 21:58:17.316075 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.316022    2582 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:17.316075 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.316039    2582 factory.go:103] Registering Raw factory
Sep 17 21:58:17.316075 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.316052    2582 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:17.316650 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.316633    2582 manager.go:349] Starting recovery of all containers
Sep 17 21:58:17.317771 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.317748    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:58:17.317875 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.317842    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:58:17.375836 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.375602    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkcw2"
Sep 17 21:58:17.376013 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.375994    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkcw2"
Sep 17 21:58:17.383040 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.383020    2582 manager.go:354] Recovery completed
Sep 17 21:58:17.387558 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.387537    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.389925 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.389908    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.389995 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.389940    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.389995 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.389950    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.390495 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.390481    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:17.390586 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.390496    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:17.390586 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.390511    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:17.390586 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.390489    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.391635 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.391618    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.391741 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.391654    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.391741 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.391665    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.392429 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.392417    2582 policy_none.go:49] "None policy: Start"
Sep 17 21:58:17.392488 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.392436    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:17.392488 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.392449    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:17.394408 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.394393    2582 policy_none.go:47] "Start"
Sep 17 21:58:17.414959 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.414941    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.426170 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426152    2582 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.426191    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426202    2582 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426450    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426469    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426575    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426662    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.426670    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.427237    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:17.439125 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.427296    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.465862 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.465826    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:17.467067 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.467050    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:17.467136 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.467102    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:17.467136 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.467124    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:17.467227 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.467169    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:17.469388 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.469365    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:17.524059 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.523985    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-9.ec2.internal\" not found" node="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.527083 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.527072    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.528225 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.528208    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.528341 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.528237    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.528341 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.528247    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.528341 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.528269    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.537401 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.537385    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.537401 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.537406    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-9.ec2.internal\": node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.551011 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.550986    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.567590 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.567560    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"]
Sep 17 21:58:17.567709 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.567632    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.569810 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.569795    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.569887 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.569822    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.569887 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.569832    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.572093 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.572081    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.572229 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.572204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.572272 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.572240    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.573102 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573085    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.573163 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573120    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.573163 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573131    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.573222 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573087    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.573222 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573187    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.573222 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.573196    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.575241 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.575227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.575293 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.575257    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:17.576131 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.576117    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:17.576207 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.576145    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:17.576207 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.576156    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:17.591161 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.591136    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-9.ec2.internal\" not found" node="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.594789 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.594770    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-9.ec2.internal\" not found" node="ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.617278 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.617248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.617278 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.617279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49246e01e0be5af303c7318bdf33cbba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.ec2.internal\" (UID: \"49246e01e0be5af303c7318bdf33cbba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.617449 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.617295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.651973 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.651915    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.718380 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49246e01e0be5af303c7318bdf33cbba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.ec2.internal\" (UID: \"49246e01e0be5af303c7318bdf33cbba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.718380 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.718629 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.718629 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.718629 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49246e01e0be5af303c7318bdf33cbba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.ec2.internal\" (UID: \"49246e01e0be5af303c7318bdf33cbba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.718629 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.718452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2da7ecbba9d989d7b60faaa4424f19b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal\" (UID: \"2da7ecbba9d989d7b60faaa4424f19b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.752467 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.752434    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.853314 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.853285    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:17.894481 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.894446    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.897937 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:17.897913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:17.953778 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:17.953739    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.054299 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.054267    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.154795 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.154730    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.188504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.188473    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:18.255711 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.255681    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.262838 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.262819    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:18.262971 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.262953    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:18.263028 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.262987    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:18.356692 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.356660    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.381853 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.381802    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:17 +0000 UTC" deadline="2028-06-02 02:52:32.61992503 +0000 UTC"
Sep 17 21:58:18.381853 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.381846    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14956h54m14.238082673s"
Sep 17 21:58:18.391890 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.391865    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:18.457816 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.457744    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.553215 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:18.553174    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49246e01e0be5af303c7318bdf33cbba.slice/crio-a60dbffb1c90ee290d3c68a54e06ef2333081ac19b617bfb1acf17aa8002f837 WatchSource:0}: Error finding container a60dbffb1c90ee290d3c68a54e06ef2333081ac19b617bfb1acf17aa8002f837: Status 404 returned error can't find the container with id a60dbffb1c90ee290d3c68a54e06ef2333081ac19b617bfb1acf17aa8002f837
Sep 17 21:58:18.553410 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:18.553391    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da7ecbba9d989d7b60faaa4424f19b3.slice/crio-0a16c34d5943e6643b0557bfbbe1d42f0319f0dd49caa71bc886b10bf13877fe WatchSource:0}: Error finding container 0a16c34d5943e6643b0557bfbbe1d42f0319f0dd49caa71bc886b10bf13877fe: Status 404 returned error can't find the container with id 0a16c34d5943e6643b0557bfbbe1d42f0319f0dd49caa71bc886b10bf13877fe
Sep 17 21:58:18.558029 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.558012    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.558131 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.558115    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:18.659022 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.658984    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.759648 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:18.759567    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-9.ec2.internal\" not found"
Sep 17 21:58:18.836382 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.836352    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:18.914857 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.914814    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:18.925197 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.925170    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:18.926106 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.926089    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal"
Sep 17 21:58:18.934929 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:18.934911    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:19.179973 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.179942    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:19.292999 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.292953    2582 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:19.301381 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.301354    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:19.301815 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.301787    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-st4hm","openshift-image-registry/node-ca-lzzd9","openshift-multus/multus-additional-cni-plugins-w2tk2","openshift-multus/network-metrics-daemon-24p4h","openshift-network-diagnostics/network-check-target-jkjz9","openshift-ovn-kubernetes/ovnkube-node-5pjh7","kube-system/konnectivity-agent-chhxg","kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9","openshift-cluster-node-tuning-operator/tuned-sq789","openshift-dns/node-resolver-j7fqc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal","openshift-multus/multus-vjwsx","openshift-network-operator/iptables-alerter-qx7qc"]
Sep 17 21:58:19.304460 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.304432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.306781 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.306758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.307339 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.307317    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:19.307536 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.307522    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:19.307828 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.307647    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9pn66\""
Sep 17 21:58:19.309211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.309182    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.309211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.309359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gkfll\""
Sep 17 21:58:19.309211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.309496    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:19.309211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.309529    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.310219 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.309711    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:19.310219 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.310069    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.311355 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.311335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.311447 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.311426    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:19.312799 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.312704    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mjbn\""
Sep 17 21:58:19.312905 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.312847    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:19.313084 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.313066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:19.313969 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.313660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:19.313969 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.313818    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:19.316164 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.315394    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:19.316164 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.316105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.320348 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.319655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.320348 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.319843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.320348 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.320287    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.320570 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.320463    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:19.321134 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.321114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vcm6l\""
Sep 17 21:58:19.321243 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.321114    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.321980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.322229    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.322236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.322304    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.322392    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:19.322504 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.322395    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.323737 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.323719    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.323827 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.323776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gch4l\""
Sep 17 21:58:19.325198 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.324841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgf5l\" (UniqueName: \"kubernetes.io/projected/cfbff581-3aee-4f8c-89e8-e47b010af5fd-kube-api-access-sgf5l\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-ovn\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-log-socket\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-k8s-cni-cncf-io\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-kubelet\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-etc-kubernetes\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-systemd-units\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-slash\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-node-log\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-env-overrides\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cnibin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.326703 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-etc-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-bin\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-config\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8bt\" (UniqueName: \"kubernetes.io/projected/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-kube-api-access-fs8bt\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-bin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7fe7daa-f235-4567-afaa-f31ed3614839-ovn-node-metrics-cert\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6ns\" (UniqueName: \"kubernetes.io/projected/c7fe7daa-f235-4567-afaa-f31ed3614839-kube-api-access-jx6ns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.326986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-system-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-multus\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-hostroot\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:19.327489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-netns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/568d748e-e14b-4c0c-b275-3807424561b2-agent-certs\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cnibin\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-kubelet\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-var-lib-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-netd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-script-lib\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-netns\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824r9\" (UniqueName: \"kubernetes.io/projected/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-kube-api-access-824r9\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327598    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2nzwv\""
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327659    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-system-cni-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/568d748e-e14b-4c0c-b275-3807424561b2-konnectivity-ca\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327759    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cni-binary-copy\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327782    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.328262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-multus-certs\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-systemd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-socket-dir-parent\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-conf-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.329098 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.327960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-daemon-config\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.329952 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.329569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.330364 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.330342    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.330465 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.330367    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.330465 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.330346    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4prkc\""
Sep 17 21:58:19.331882 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.331865    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.331986 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.331969    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.332133 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.332117    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nlfrw\""
Sep 17 21:58:19.332803 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.332784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.335236 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.335209    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:19.336225 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.336107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:19.336371 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.336357    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:19.336371 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.336369    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-68fdc\""
Sep 17 21:58:19.347048 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.347023    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjj2g"
Sep 17 21:58:19.354375 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.354338    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjj2g"
Sep 17 21:58:19.416260 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.416219    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:19.428174 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cni-binary-copy\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-multus-certs\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-systemd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1330de1-e458-48dd-87f7-0376f228c47f-hosts-file\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39743950-452b-46ba-95ce-b69a6ea9c790-host\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysconfig\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-kubernetes\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.428332 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-k8s-cni-cncf-io\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-etc-kubernetes\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-systemd-units\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-slash\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-env-overrides\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-systemd\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-etc-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-bin\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-config\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-run\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-slash\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8bt\" (UniqueName: \"kubernetes.io/projected/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-kube-api-access-fs8bt\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-bin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-multus-certs\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.428873 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-bin\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6ns\" (UniqueName: \"kubernetes.io/projected/c7fe7daa-f235-4567-afaa-f31ed3614839-kube-api-access-jx6ns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-system-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-multus\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-hostroot\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.428765    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-dbus\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b58dc0db-afc2-4c56-9f65-161b881d7266-host-slash\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-registration-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cni-binary-copy\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-hostroot\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-systemd-units\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-multus\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.428981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-system-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.429050    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:19.928826626 +0000 UTC m=+4.596889354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:19.429679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-etc-kubernetes\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-env-overrides\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-config\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-etc-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-k8s-cni-cncf-io\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-systemd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-cni-bin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-var-lib-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-netd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-script-lib\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-cni-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-cni-netd\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-kubelet-config\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429457    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-var-lib-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.430573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c1330de1-e458-48dd-87f7-0376f228c47f-kube-api-access-nmmz6\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b58dc0db-afc2-4c56-9f65-161b881d7266-iptables-alerter-script\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-netns\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824r9\" (UniqueName: \"kubernetes.io/projected/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-kube-api-access-824r9\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-run-netns\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-socket-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdfz\" (UniqueName: \"kubernetes.io/projected/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kube-api-access-dcdfz\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-socket-dir-parent\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-conf-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-daemon-config\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf5l\" (UniqueName: \"kubernetes.io/projected/cfbff581-3aee-4f8c-89e8-e47b010af5fd-kube-api-access-sgf5l\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-socket-dir-parent\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-ovn\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.431442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-conf-dir\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-log-socket\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7fe7daa-f235-4567-afaa-f31ed3614839-ovnkube-script-lib\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-ovn\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.429987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-log-socket\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-kubelet\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430112    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-node-log\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-tuned\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-host-var-lib-kubelet\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-node-log\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-tmp\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cnibin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-multus-daemon-config\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-cnibin\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.432285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-conf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-lib-modules\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7fe7daa-f235-4567-afaa-f31ed3614839-ovn-node-metrics-cert\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-run-openvswitch\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-modprobe-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-host\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqnf\" (UniqueName: \"kubernetes.io/projected/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-kube-api-access-hzqnf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-netns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-device-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-sys-fs\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/568d748e-e14b-4c0c-b275-3807424561b2-agent-certs\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-run-netns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cnibin\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-kubelet\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-cnibin\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.433085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430815    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrttq\" (UniqueName: \"kubernetes.io/projected/b58dc0db-afc2-4c56-9f65-161b881d7266-kube-api-access-zrttq\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7fe7daa-f235-4567-afaa-f31ed3614839-host-kubelet\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-system-cni-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1330de1-e458-48dd-87f7-0376f228c47f-tmp-dir\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-system-cni-dir\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.430963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39743950-452b-46ba-95ce-b69a6ea9c790-serviceca\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.431038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgmz\" (UniqueName: \"kubernetes.io/projected/39743950-452b-46ba-95ce-b69a6ea9c790-kube-api-access-fvgmz\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.431062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-sys\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.431083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-var-lib-kubelet\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.431118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/568d748e-e14b-4c0c-b275-3807424561b2-konnectivity-ca\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.433945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.431614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/568d748e-e14b-4c0c-b275-3807424561b2-konnectivity-ca\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.434727 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.434668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7fe7daa-f235-4567-afaa-f31ed3614839-ovn-node-metrics-cert\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.434841 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.434783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/568d748e-e14b-4c0c-b275-3807424561b2-agent-certs\") pod \"konnectivity-agent-chhxg\" (UID: \"568d748e-e14b-4c0c-b275-3807424561b2\") " pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.439531 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.439506    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:19.439531 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.439533    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:19.439531 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.439563    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:19.439765 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.439632    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:19.939614247 +0000 UTC m=+4.607677005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:19.440231 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.440211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6ns\" (UniqueName: \"kubernetes.io/projected/c7fe7daa-f235-4567-afaa-f31ed3614839-kube-api-access-jx6ns\") pod \"ovnkube-node-5pjh7\" (UID: \"c7fe7daa-f235-4567-afaa-f31ed3614839\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.441010 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.440986    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8bt\" (UniqueName: \"kubernetes.io/projected/e1b685a8-ec3e-4a39-92f6-deb21d8dba13-kube-api-access-fs8bt\") pod \"multus-additional-cni-plugins-w2tk2\" (UID: \"e1b685a8-ec3e-4a39-92f6-deb21d8dba13\") " pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.441357 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.441336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824r9\" (UniqueName: \"kubernetes.io/projected/f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f-kube-api-access-824r9\") pod \"multus-vjwsx\" (UID: \"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f\") " pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.442121 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.442101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgf5l\" (UniqueName: \"kubernetes.io/projected/cfbff581-3aee-4f8c-89e8-e47b010af5fd-kube-api-access-sgf5l\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.472464 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.472409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal" event={"ID":"2da7ecbba9d989d7b60faaa4424f19b3","Type":"ContainerStarted","Data":"0a16c34d5943e6643b0557bfbbe1d42f0319f0dd49caa71bc886b10bf13877fe"}
Sep 17 21:58:19.473619 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.473582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal" event={"ID":"49246e01e0be5af303c7318bdf33cbba","Type":"ContainerStarted","Data":"a60dbffb1c90ee290d3c68a54e06ef2333081ac19b617bfb1acf17aa8002f837"}
Sep 17 21:58:19.531365 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-run\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-dbus\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b58dc0db-afc2-4c56-9f65-161b881d7266-host-slash\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-registration-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-kubelet-config\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c1330de1-e458-48dd-87f7-0376f228c47f-kube-api-access-nmmz6\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b58dc0db-afc2-4c56-9f65-161b881d7266-iptables-alerter-script\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-registration-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531562 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b58dc0db-afc2-4c56-9f65-161b881d7266-host-slash\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-kubelet-config\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-socket-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3397b63b-b850-4a86-9411-754f199657fe-dbus\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdfz\" (UniqueName: \"kubernetes.io/projected/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kube-api-access-dcdfz\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-tuned\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-tmp\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-conf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-lib-modules\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-modprobe-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-host\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqnf\" (UniqueName: \"kubernetes.io/projected/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-kube-api-access-hzqnf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-device-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-socket-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.531957 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-sys-fs\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-run\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.531938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.532048    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-modprobe-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b58dc0db-afc2-4c56-9f65-161b881d7266-iptables-alerter-script\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.532106    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:20.032087625 +0000 UTC m=+4.700150364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-host\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-conf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrttq\" (UniqueName: \"kubernetes.io/projected/b58dc0db-afc2-4c56-9f65-161b881d7266-kube-api-access-zrttq\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1330de1-e458-48dd-87f7-0376f228c47f-tmp-dir\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysctl-d\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39743950-452b-46ba-95ce-b69a6ea9c790-serviceca\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgmz\" (UniqueName: \"kubernetes.io/projected/39743950-452b-46ba-95ce-b69a6ea9c790-kube-api-access-fvgmz\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-sys-fs\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-sys\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.532757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-var-lib-kubelet\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1330de1-e458-48dd-87f7-0376f228c47f-hosts-file\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39743950-452b-46ba-95ce-b69a6ea9c790-host\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysconfig\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-sys\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-kubernetes\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1330de1-e458-48dd-87f7-0376f228c47f-hosts-file\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-systemd\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39743950-452b-46ba-95ce-b69a6ea9c790-serviceca\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532666    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-sysconfig\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-kubernetes\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-systemd\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39743950-452b-46ba-95ce-b69a6ea9c790-host\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-var-lib-kubelet\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.532873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d1d518a-9d5d-4175-9cbf-500a2ece005c-device-dir\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.533026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-lib-modules\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.533490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.533085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1330de1-e458-48dd-87f7-0376f228c47f-tmp-dir\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.534312 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.534293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-etc-tuned\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.534802 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.534777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-tmp\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.540646 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.540622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdfz\" (UniqueName: \"kubernetes.io/projected/6d1d518a-9d5d-4175-9cbf-500a2ece005c-kube-api-access-dcdfz\") pod \"aws-ebs-csi-driver-node-hjzs9\" (UID: \"6d1d518a-9d5d-4175-9cbf-500a2ece005c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.540767 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.540745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqnf\" (UniqueName: \"kubernetes.io/projected/d7eaf875-1ad0-4a74-86d2-b7432d69d44d-kube-api-access-hzqnf\") pod \"tuned-sq789\" (UID: \"d7eaf875-1ad0-4a74-86d2-b7432d69d44d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.540858 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.540825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgmz\" (UniqueName: \"kubernetes.io/projected/39743950-452b-46ba-95ce-b69a6ea9c790-kube-api-access-fvgmz\") pod \"node-ca-lzzd9\" (UID: \"39743950-452b-46ba-95ce-b69a6ea9c790\") " pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.540918 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.540880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrttq\" (UniqueName: \"kubernetes.io/projected/b58dc0db-afc2-4c56-9f65-161b881d7266-kube-api-access-zrttq\") pod \"iptables-alerter-qx7qc\" (UID: \"b58dc0db-afc2-4c56-9f65-161b881d7266\") " pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.541111 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.541092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c1330de1-e458-48dd-87f7-0376f228c47f-kube-api-access-nmmz6\") pod \"node-resolver-j7fqc\" (UID: \"c1330de1-e458-48dd-87f7-0376f228c47f\") " pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.618316 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.618281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:19.626687 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.626661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vjwsx"
Sep 17 21:58:19.636540 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.636517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2tk2"
Sep 17 21:58:19.644242 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.644220    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq789"
Sep 17 21:58:19.651093 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.651073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:19.659785 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.659762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9"
Sep 17 21:58:19.666655 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.666630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lzzd9"
Sep 17 21:58:19.682180 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.682127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7fqc"
Sep 17 21:58:19.688792 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.688771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qx7qc"
Sep 17 21:58:19.935848 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:19.935759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:19.935997 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.935949    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:19.936047 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:19.936031    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:20.936010119 +0000 UTC m=+5.604072858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:20.036951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.036918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:20.036951 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.036958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:20.037127 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037063    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:20.037127 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037116    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:21.037103512 +0000 UTC m=+5.705166236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:20.037195 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037114    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:20.037195 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037144    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:20.037195 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037157    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:20.037287 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.037224    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:21.037205864 +0000 UTC m=+5.705268593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:20.340995 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.340965    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:20.355813 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.355788    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b685a8_ec3e_4a39_92f6_deb21d8dba13.slice/crio-a00c085c8c665dbb0f8c398b70b8ca05ea7b75d5d09c3ae0a70b7a8ab3def3e8 WatchSource:0}: Error finding container a00c085c8c665dbb0f8c398b70b8ca05ea7b75d5d09c3ae0a70b7a8ab3def3e8: Status 404 returned error can't find the container with id a00c085c8c665dbb0f8c398b70b8ca05ea7b75d5d09c3ae0a70b7a8ab3def3e8
Sep 17 21:58:20.357125 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.357095    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1330de1_e458_48dd_87f7_0376f228c47f.slice/crio-47b91b18b5e5a0995b6b7aadaa1a5864961cd53d22947d138c00fe3ab740e0bf WatchSource:0}: Error finding container 47b91b18b5e5a0995b6b7aadaa1a5864961cd53d22947d138c00fe3ab740e0bf: Status 404 returned error can't find the container with id 47b91b18b5e5a0995b6b7aadaa1a5864961cd53d22947d138c00fe3ab740e0bf
Sep 17 21:58:20.357228 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.357124    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:19 +0000 UTC" deadline="2028-04-10 15:25:23.395133904 +0000 UTC"
Sep 17 21:58:20.357228 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.357166    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13697h27m3.037970919s"
Sep 17 21:58:20.358618 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.358593    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fe7daa_f235_4567_afaa_f31ed3614839.slice/crio-787eb6739caddf7a53094d1a009bf097c664b624859f9d6638fe0371ce28b6b8 WatchSource:0}: Error finding container 787eb6739caddf7a53094d1a009bf097c664b624859f9d6638fe0371ce28b6b8: Status 404 returned error can't find the container with id 787eb6739caddf7a53094d1a009bf097c664b624859f9d6638fe0371ce28b6b8
Sep 17 21:58:20.359683 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.359648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1d518a_9d5d_4175_9cbf_500a2ece005c.slice/crio-8a1334b39ebb78dca9cd1cf99fd63f64cf151a292a6cc662db5ed97835fb2334 WatchSource:0}: Error finding container 8a1334b39ebb78dca9cd1cf99fd63f64cf151a292a6cc662db5ed97835fb2334: Status 404 returned error can't find the container with id 8a1334b39ebb78dca9cd1cf99fd63f64cf151a292a6cc662db5ed97835fb2334
Sep 17 21:58:20.361134 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.360912    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eaf875_1ad0_4a74_86d2_b7432d69d44d.slice/crio-9220e35dfecdde8d47410b4f5e75eb4a3c7310714bcc1601dc40df2322e464fe WatchSource:0}: Error finding container 9220e35dfecdde8d47410b4f5e75eb4a3c7310714bcc1601dc40df2322e464fe: Status 404 returned error can't find the container with id 9220e35dfecdde8d47410b4f5e75eb4a3c7310714bcc1601dc40df2322e464fe
Sep 17 21:58:20.362431 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.362407    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39743950_452b_46ba_95ce_b69a6ea9c790.slice/crio-82a2df6d4ea2de323c4778a183fa87c55a18ee5a9c28a181149651a79f38be70 WatchSource:0}: Error finding container 82a2df6d4ea2de323c4778a183fa87c55a18ee5a9c28a181149651a79f38be70: Status 404 returned error can't find the container with id 82a2df6d4ea2de323c4778a183fa87c55a18ee5a9c28a181149651a79f38be70
Sep 17 21:58:20.385631 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.385603    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568d748e_e14b_4c0c_b275_3807424561b2.slice/crio-6d9713d486fe77c56d450d214f467038925596d45cca6df4c65791a24aced9cb WatchSource:0}: Error finding container 6d9713d486fe77c56d450d214f467038925596d45cca6df4c65791a24aced9cb: Status 404 returned error can't find the container with id 6d9713d486fe77c56d450d214f467038925596d45cca6df4c65791a24aced9cb
Sep 17 21:58:20.385856 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.385839    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a63491_4f67_4d45_a7bc_1a5aa5a1b26f.slice/crio-be97e5b1f4659bcb6029e5e9485f25be1bc26fdc68f336cdb678654dce8bbc31 WatchSource:0}: Error finding container be97e5b1f4659bcb6029e5e9485f25be1bc26fdc68f336cdb678654dce8bbc31: Status 404 returned error can't find the container with id be97e5b1f4659bcb6029e5e9485f25be1bc26fdc68f336cdb678654dce8bbc31
Sep 17 21:58:20.388356 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:20.388171    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58dc0db_afc2_4c56_9f65_161b881d7266.slice/crio-6b3d52c900cbbbbeff146d5df608511b863204936284377185f8a88e43b507c1 WatchSource:0}: Error finding container 6b3d52c900cbbbbeff146d5df608511b863204936284377185f8a88e43b507c1: Status 404 returned error can't find the container with id 6b3d52c900cbbbbeff146d5df608511b863204936284377185f8a88e43b507c1
Sep 17 21:58:20.467949 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.467922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:20.468087 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.468047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:20.476578 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.476531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qx7qc" event={"ID":"b58dc0db-afc2-4c56-9f65-161b881d7266","Type":"ContainerStarted","Data":"6b3d52c900cbbbbeff146d5df608511b863204936284377185f8a88e43b507c1"}
Sep 17 21:58:20.477710 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.477681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjwsx" event={"ID":"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f","Type":"ContainerStarted","Data":"be97e5b1f4659bcb6029e5e9485f25be1bc26fdc68f336cdb678654dce8bbc31"}
Sep 17 21:58:20.479100 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.479066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chhxg" event={"ID":"568d748e-e14b-4c0c-b275-3807424561b2","Type":"ContainerStarted","Data":"6d9713d486fe77c56d450d214f467038925596d45cca6df4c65791a24aced9cb"}
Sep 17 21:58:20.480299 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.480270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq789" event={"ID":"d7eaf875-1ad0-4a74-86d2-b7432d69d44d","Type":"ContainerStarted","Data":"9220e35dfecdde8d47410b4f5e75eb4a3c7310714bcc1601dc40df2322e464fe"}
Sep 17 21:58:20.481421 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.481399    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9" event={"ID":"6d1d518a-9d5d-4175-9cbf-500a2ece005c","Type":"ContainerStarted","Data":"8a1334b39ebb78dca9cd1cf99fd63f64cf151a292a6cc662db5ed97835fb2334"}
Sep 17 21:58:20.482520 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.482498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"787eb6739caddf7a53094d1a009bf097c664b624859f9d6638fe0371ce28b6b8"}
Sep 17 21:58:20.483689 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.483659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerStarted","Data":"a00c085c8c665dbb0f8c398b70b8ca05ea7b75d5d09c3ae0a70b7a8ab3def3e8"}
Sep 17 21:58:20.485714 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.485674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal" event={"ID":"49246e01e0be5af303c7318bdf33cbba","Type":"ContainerStarted","Data":"a9a9536eaf7fd5b4d0da6db9d498c9414b404d7ad8fd9155f9c39b108d9c5e55"}
Sep 17 21:58:20.487085 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.487061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lzzd9" event={"ID":"39743950-452b-46ba-95ce-b69a6ea9c790","Type":"ContainerStarted","Data":"82a2df6d4ea2de323c4778a183fa87c55a18ee5a9c28a181149651a79f38be70"}
Sep 17 21:58:20.488284 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.488264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7fqc" event={"ID":"c1330de1-e458-48dd-87f7-0376f228c47f","Type":"ContainerStarted","Data":"47b91b18b5e5a0995b6b7aadaa1a5864961cd53d22947d138c00fe3ab740e0bf"}
Sep 17 21:58:20.499952 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.499912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.ec2.internal" podStartSLOduration=2.499902025 podStartE2EDuration="2.499902025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:20.499601946 +0000 UTC m=+5.167664718" watchObservedRunningTime="2026-09-17 21:58:20.499902025 +0000 UTC m=+5.167964772"
Sep 17 21:58:20.944196 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:20.943606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:20.944196 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.943747    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:20.944196 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:20.943811    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.943793 +0000 UTC m=+7.611855746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.044417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.044520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.044668    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.044685    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.044698    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.044755    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.044737154 +0000 UTC m=+7.712799891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.045150    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.045292 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.045198    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.045183804 +0000 UTC m=+7.713246535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.358385 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.358343    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:19 +0000 UTC" deadline="2028-04-16 09:43:48.723535162 +0000 UTC"
Sep 17 21:58:21.358385 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.358384    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13835h45m27.365156645s"
Sep 17 21:58:21.468211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.467669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:21.468211 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.467681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:21.468211 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.467801    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:21.468211 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:21.467901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:21.501573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.501089    2582 generic.go:356] "Generic (PLEG): container finished" podID="2da7ecbba9d989d7b60faaa4424f19b3" containerID="7f5e89cfd740edbe768ced61d6b92fda8079ce6ddd5c76f283197dd9df78ee94" exitCode=0
Sep 17 21:58:21.501752 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:21.501614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal" event={"ID":"2da7ecbba9d989d7b60faaa4424f19b3","Type":"ContainerDied","Data":"7f5e89cfd740edbe768ced61d6b92fda8079ce6ddd5c76f283197dd9df78ee94"}
Sep 17 21:58:22.468026 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:22.467484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:22.468026 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:22.467650    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:22.514307 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:22.513525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal" event={"ID":"2da7ecbba9d989d7b60faaa4424f19b3","Type":"ContainerStarted","Data":"61757a7011dbe785721b54796a7e7e9f56a2c8298beb5b34aee263bc19b73abe"}
Sep 17 21:58:22.533273 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:22.532469    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.ec2.internal" podStartSLOduration=4.53244887 podStartE2EDuration="4.53244887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:22.531259508 +0000 UTC m=+7.199322255" watchObservedRunningTime="2026-09-17 21:58:22.53244887 +0000 UTC m=+7.200511617"
Sep 17 21:58:22.965582 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:22.964955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:22.965582 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:22.965130    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:22.965582 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:22.965197    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.96517798 +0000 UTC m=+11.633240712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:23.065813 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:23.065774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:23.065996 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:23.065836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:23.065996 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.065980    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.066101 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.066039    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:27.066022289 +0000 UTC m=+11.734085018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.066477 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.066432    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:23.066477 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.066459    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:23.066477 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.066473    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.066686 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.066516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:27.066502174 +0000 UTC m=+11.734564914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.467705 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:23.467666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:23.467900 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.467796    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:23.467900 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:23.467678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:23.468006 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:23.467895    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:24.467678 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:24.467500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:24.467678 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:24.467663    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:25.468137 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:25.467507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:25.468137 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:25.467660    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:25.468137 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:25.468036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:25.468137 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:25.468139    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:26.467435 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:26.467374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:26.467652 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:26.467529    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:27.002569 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:26.996221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:27.003289 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.003252    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:27.003424 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.003352    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:35.003330956 +0000 UTC m=+19.671393704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:27.097783 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:27.097720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:27.097783 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:27.097786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:27.098028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.097924    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:27.098028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.097960    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:27.098028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.097990    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:27.098028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.098006    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:27.098224 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.097994    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:35.097975552 +0000 UTC m=+19.766038287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:27.098224 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.098072    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:35.098054606 +0000 UTC m=+19.766117337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:27.469063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:27.468460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:27.469063 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.468588    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:27.469063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:27.468661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:27.469063 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:27.468736    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:28.468269 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:28.468196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:28.468746 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:28.468341    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:29.467984 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:29.467953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:29.467984 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:29.467974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:29.468226 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:29.468091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:29.468287 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:29.468245    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:30.467929 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:30.467890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:30.468409 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:30.468045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:31.468002 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:31.467916    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:31.468002 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:31.467955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:31.468462 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:31.468132    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:31.468462 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:31.468268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:32.467711 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:32.467671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:32.467881 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:32.467808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:33.468363 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:33.468332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:33.468843 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:33.468369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:33.468843 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:33.468473    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:33.468843 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:33.468627    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:34.467702 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:34.467670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:34.467857 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:34.467820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:35.051528 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:35.051490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:35.051999 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.051678    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:35.051999 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.051758    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:51.051739088 +0000 UTC m=+35.719801832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:35.152661 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:35.152621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:35.152834 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:35.152718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:35.152834 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152778    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:35.152834 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152818    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:35.152834 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152832    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:35.153028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152844    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:35.153028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152845    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:58:51.152824546 +0000 UTC m=+35.820887284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:35.153028 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.152887    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:51.152875645 +0000 UTC m=+35.820938374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:35.467457 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:35.467419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:35.467644 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:35.467421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:35.467644 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.467577    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:35.467742 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:35.467645    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:36.467460 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:36.467421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:36.467887 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:36.467577    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:37.468913 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:37.468881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:37.469326 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:37.468962    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:37.469326 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:37.469054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:37.469326 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:37.469172    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:38.467995 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.467665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:38.468156 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:38.468120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:38.549128 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.549016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7fqc" event={"ID":"c1330de1-e458-48dd-87f7-0376f228c47f","Type":"ContainerStarted","Data":"494522e8e0657ac8b79ceb57a11f3bb03584c34fbf789660f8ea7103cfb6b49d"}
Sep 17 21:58:38.551079 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.551050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vjwsx" event={"ID":"f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f","Type":"ContainerStarted","Data":"c1f05259eddc83df186a26037ede8251d66c409ea688fdab3490766bd5418d29"}
Sep 17 21:58:38.553305 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.553254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chhxg" event={"ID":"568d748e-e14b-4c0c-b275-3807424561b2","Type":"ContainerStarted","Data":"9e68d8e1d9abaaae8a07bc0489c1bf6ef3f9c2f8255560a08aab8955e6c3b977"}
Sep 17 21:58:38.555092 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.555071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq789" event={"ID":"d7eaf875-1ad0-4a74-86d2-b7432d69d44d","Type":"ContainerStarted","Data":"155c0c14059a003f25a4b94f38584660c49b74792ee7dd2e3d98ef0ba1ead5bf"}
Sep 17 21:58:38.556718 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.556697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9" event={"ID":"6d1d518a-9d5d-4175-9cbf-500a2ece005c","Type":"ContainerStarted","Data":"9f44b5fe5bf63ace79d811d0a4ae7e51cb65a9cca4f6a1808a3891a57bea87a3"}
Sep 17 21:58:38.560056 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"3f21bf4673b5dd09eeba4ae70de2b53d86618b3055936f766ec0b15ffb6225b1"}
Sep 17 21:58:38.560145 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"66d9d5c202413a734daa9aad9795f2d666a7a87af3cd2952949ad77a2e46eda6"}
Sep 17 21:58:38.560215 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"56caf88f93331a6e5c6cd50c01a81b6201878ec9778a13d874bf41099d897b34"}
Sep 17 21:58:38.560215 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"71f4cd958da26461d5278b1ba59ea0f01183352843c70d48d09bc92ca7c0ef25"}
Sep 17 21:58:38.560320 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"af67a1518307abf53a30dbff8f21b8f2fe84ed2ad1e37e60549339360f56a83f"}
Sep 17 21:58:38.560320 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.560272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"c1953d027f23a78c40da42a078eb15acdd5a1d0723e11151fbec03f5cead9ad1"}
Sep 17 21:58:38.561825 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.561801    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="61f61730b6fd3d57391f8d69447c320cda08ce6d8e43631bb6775b402bf79b94" exitCode=0
Sep 17 21:58:38.561914 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.561889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"61f61730b6fd3d57391f8d69447c320cda08ce6d8e43631bb6775b402bf79b94"}
Sep 17 21:58:38.563606 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.563517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lzzd9" event={"ID":"39743950-452b-46ba-95ce-b69a6ea9c790","Type":"ContainerStarted","Data":"9130d44c27436f2b8e38d9a91dea080fad98dda18c2aef4cfc3744b31e8731c5"}
Sep 17 21:58:38.564513 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.564468    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7fqc" podStartSLOduration=4.193803941 podStartE2EDuration="21.56445251s" totalImagesPullingTime="17.370648569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.563772658 +0000 UTC m=+23.231835422" watchObservedRunningTime="2026-09-17 21:58:38.56445251 +0000 UTC m=+23.232515259"
Sep 17 21:58:38.599975 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.598461    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lzzd9" podStartSLOduration=4.222025348 podStartE2EDuration="21.598444639s" totalImagesPullingTime="17.376419291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.597657319 +0000 UTC m=+23.265720083" watchObservedRunningTime="2026-09-17 21:58:38.598444639 +0000 UTC m=+23.266507385"
Sep 17 21:58:38.617382 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.617335    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sq789" podStartSLOduration=4.240983449 podStartE2EDuration="21.617322102s" totalImagesPullingTime="17.376338653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.616907994 +0000 UTC m=+23.284970760" watchObservedRunningTime="2026-09-17 21:58:38.617322102 +0000 UTC m=+23.285384849"
Sep 17 21:58:38.632694 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.632652    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-chhxg" podStartSLOduration=11.73213947 podStartE2EDuration="21.632637054s" totalImagesPullingTime="9.900497584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.632421341 +0000 UTC m=+23.300484088" watchObservedRunningTime="2026-09-17 21:58:38.632637054 +0000 UTC m=+23.300699801"
Sep 17 21:58:38.649704 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.649651    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vjwsx" podStartSLOduration=4.233005501 podStartE2EDuration="21.649632153s" totalImagesPullingTime="17.416626652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.649319057 +0000 UTC m=+23.317381827" watchObservedRunningTime="2026-09-17 21:58:38.649632153 +0000 UTC m=+23.317694901"
Sep 17 21:58:38.951556 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:38.951521    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:39.439114 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.439006    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:38.951537636Z","UUID":"9d54f590-a83a-4eda-b02d-9a2b5dbfbab0","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:39.440980 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.440949    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:39.440980 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.440986    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:39.467397 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.467368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:39.467573 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.467408    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:39.467573 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:39.467498    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:39.467693 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:39.467656    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:39.527368 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.527306    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:39.527879 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.527863    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:39.568124 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.568089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qx7qc" event={"ID":"b58dc0db-afc2-4c56-9f65-161b881d7266","Type":"ContainerStarted","Data":"b8e8b133f51889f2f40d611112e32e428d359b62b9f996af553e33d401bee2b9"}
Sep 17 21:58:39.570353 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.570317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9" event={"ID":"6d1d518a-9d5d-4175-9cbf-500a2ece005c","Type":"ContainerStarted","Data":"a507656701677789ae654197ff4a5eb2d152ba1154279c8a3d6ab79986528cd1"}
Sep 17 21:58:39.582811 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:39.582757    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qx7qc" podStartSLOduration=5.24363439 podStartE2EDuration="22.582738981s" totalImagesPullingTime="17.339104591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:39.582298745 +0000 UTC m=+24.250361523" watchObservedRunningTime="2026-09-17 21:58:39.582738981 +0000 UTC m=+24.250801729"
Sep 17 21:58:40.468024 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:40.467841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:40.468170 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:40.468120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:40.575713 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:40.575677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9" event={"ID":"6d1d518a-9d5d-4175-9cbf-500a2ece005c","Type":"ContainerStarted","Data":"ac088d2d8767f58191a5ace719cc73159cfd74d63d22825ee95cbe85ab717b5f"}
Sep 17 21:58:40.579893 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:40.579858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"d9e0ce50fc84d4dc6f64b0cb63bbe54fdfa7cf3d10d1805d2af8419d22b8c5d2"}
Sep 17 21:58:40.580061 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:40.579949    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:40.593718 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:40.593667    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjzs9" podStartSLOduration=4.751446952 podStartE2EDuration="23.593648658s" totalImagesPullingTime="18.842201706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:40.593405008 +0000 UTC m=+25.261467776" watchObservedRunningTime="2026-09-17 21:58:40.593648658 +0000 UTC m=+25.261711422"
Sep 17 21:58:41.467565 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:41.467521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:41.467565 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:41.467536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:41.467803 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:41.467656    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:41.467803 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:41.467757    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:42.467628 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:42.467596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:42.468125 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:42.467717    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:43.467903 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.467718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:43.468583 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.467790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:43.468583 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:43.467997    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:43.468583 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:43.468073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:43.588685 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.588654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" event={"ID":"c7fe7daa-f235-4567-afaa-f31ed3614839","Type":"ContainerStarted","Data":"692e689fb8ab25a1dba95d7c22cbfae436a40856852a758b0cd0455d8eb9d61c"}
Sep 17 21:58:43.588918 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.588900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:43.590668 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.590640    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="cbe6eb9e18df9708dd206ec031027b52d0ce58b000ba310acddf0cc9e0aa5413" exitCode=0
Sep 17 21:58:43.590773 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.590683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"cbe6eb9e18df9708dd206ec031027b52d0ce58b000ba310acddf0cc9e0aa5413"}
Sep 17 21:58:43.607020 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.606990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:43.624280 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:43.624229    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7" podStartSLOduration=9.212718255 podStartE2EDuration="26.624215093s" totalImagesPullingTime="17.411496838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:43.622475645 +0000 UTC m=+28.290538392" watchObservedRunningTime="2026-09-17 21:58:43.624215093 +0000 UTC m=+28.292277840"
Sep 17 21:58:44.467757 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.467720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:44.467953 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:44.467856    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:44.593484 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.593458    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:44.593932 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.593907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:44.609280 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.609255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:44.913349 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.913322    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:58:44.949298 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.949268    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jkjz9"]
Sep 17 21:58:44.949443 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.949378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:44.949505 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:44.949464    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:44.950734 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.950704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-st4hm"]
Sep 17 21:58:44.950868 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.950822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:44.950949 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:44.950928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:44.951311 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.951293    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24p4h"]
Sep 17 21:58:44.951381 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:44.951370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:44.951478 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:44.951459    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:45.598317 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:45.598132    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="f24187966206b90f47306862a2e92a63a3367e75d8d9e83fbfb8e596b823c239" exitCode=0
Sep 17 21:58:45.598892 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:45.598223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"f24187966206b90f47306862a2e92a63a3367e75d8d9e83fbfb8e596b823c239"}
Sep 17 21:58:46.468238 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:46.468202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:46.468434 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:46.468315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:46.468434 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:46.468329    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:46.468434 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:46.468343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:46.468434 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:46.468399    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:46.468674 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:46.468496    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:47.605416 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:47.605381    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="81ee154ad3e16e8ae1af844dbc1a03864f32ab4bbd0259f14409464cfaef62ca" exitCode=0
Sep 17 21:58:47.606063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:47.605432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"81ee154ad3e16e8ae1af844dbc1a03864f32ab4bbd0259f14409464cfaef62ca"}
Sep 17 21:58:48.467542 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:48.467511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:48.467754 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:48.467563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:48.467754 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:48.467682    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:48.467931 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:48.467910    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:48.467995 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:48.467972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:48.468076 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:48.468053    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:49.387302 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:49.387272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:49.387812 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:49.387394    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:49.388005 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:49.387981    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chhxg"
Sep 17 21:58:50.467328 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:50.467292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:50.467897 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:50.467295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:50.467897 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:50.467413    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjz9" podUID="4819ba77-53a1-4a53-9643-699d42acdff5"
Sep 17 21:58:50.467897 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:50.467294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:50.467897 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:50.467525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24p4h" podUID="cfbff581-3aee-4f8c-89e8-e47b010af5fd"
Sep 17 21:58:50.467897 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:50.467624    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-st4hm" podUID="3397b63b-b850-4a86-9411-754f199657fe"
Sep 17 21:58:51.063372 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.063331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:51.063586 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.063500    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:51.063656 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.063586    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs podName:cfbff581-3aee-4f8c-89e8-e47b010af5fd nodeName:}" failed. No retries permitted until 2026-09-17 21:59:23.063566484 +0000 UTC m=+67.731629226 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs") pod "network-metrics-daemon-24p4h" (UID: "cfbff581-3aee-4f8c-89e8-e47b010af5fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:51.164391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.164239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:51.164391 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.164364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:51.164391 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164367    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:51.164850 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164472    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret podName:3397b63b-b850-4a86-9411-754f199657fe nodeName:}" failed. No retries permitted until 2026-09-17 21:59:23.164453394 +0000 UTC m=+67.832516133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret") pod "global-pull-secret-syncer-st4hm" (UID: "3397b63b-b850-4a86-9411-754f199657fe") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:51.164850 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164530    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:51.164850 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164573    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:51.164850 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164587    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dbcqk for pod openshift-network-diagnostics/network-check-target-jkjz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:51.164850 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:51.164657    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk podName:4819ba77-53a1-4a53-9643-699d42acdff5 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:23.164642201 +0000 UTC m=+67.832704931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbcqk" (UniqueName: "kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk") pod "network-check-target-jkjz9" (UID: "4819ba77-53a1-4a53-9643-699d42acdff5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:51.178956 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.178925    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7fqc_c1330de1-e458-48dd-87f7-0376f228c47f/dns-node-resolver/0.log"
Sep 17 21:58:51.826704 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.826668    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-9.ec2.internal" event="NodeReady"
Sep 17 21:58:51.827259 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.826818    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:51.879285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.879262    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rrh9f"]
Sep 17 21:58:51.902346 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.902312    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkfpl"]
Sep 17 21:58:51.902507 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.902441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:51.905607 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.905581    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kb28p\""
Sep 17 21:58:51.906608 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.906587    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:51.907193 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.907172    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:51.918370 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.918343    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrh9f"]
Sep 17 21:58:51.918482 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.918392    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-skjpn"]
Sep 17 21:58:51.918583 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.918541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:51.921350 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.921325    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:58:51.921459 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.921349    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:58:51.921459 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.921405    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:58:51.921459 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.921448    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:58:51.921754 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.921738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qxfkf\""
Sep 17 21:58:51.936455 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.936417    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkfpl"]
Sep 17 21:58:51.936582 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.936518    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-skjpn"]
Sep 17 21:58:51.936634 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.936616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:51.939460 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.939435    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:51.939575 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.939491    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:51.939575 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.939560    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5ht6s\""
Sep 17 21:58:51.939675 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.939565    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:51.969202 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1351b34-0a6a-499c-95cf-f5c292809054-tmp-dir\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:51.969356 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:51.969356 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32d5d2b7-74a2-4e98-8344-89d03e643638-crio-socket\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:51.969356 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:51.969356 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7qw\" (UniqueName: \"kubernetes.io/projected/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-api-access-rv7qw\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:51.969356 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1351b34-0a6a-499c-95cf-f5c292809054-config-volume\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:51.969620 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zx8\" (UniqueName: \"kubernetes.io/projected/d1351b34-0a6a-499c-95cf-f5c292809054-kube-api-access-d2zx8\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:51.969620 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:51.969620 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:51.969449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d5d2b7-74a2-4e98-8344-89d03e643638-data-volume\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.070601 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d5d2b7-74a2-4e98-8344-89d03e643638-data-volume\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-cert\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1351b34-0a6a-499c-95cf-f5c292809054-tmp-dir\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32d5d2b7-74a2-4e98-8344-89d03e643638-crio-socket\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.070740 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070727    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070801    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls podName:d1351b34-0a6a-499c-95cf-f5c292809054 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.570772784 +0000 UTC m=+37.238835523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls") pod "dns-default-rrh9f" (UID: "d1351b34-0a6a-499c-95cf-f5c292809054") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070811    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070834    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070882    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.57086573 +0000 UTC m=+37.238928479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7qw\" (UniqueName: \"kubernetes.io/projected/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-api-access-rv7qw\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32d5d2b7-74a2-4e98-8344-89d03e643638-crio-socket\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.070956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1351b34-0a6a-499c-95cf-f5c292809054-config-volume\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.070966    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.570949314 +0000 UTC m=+37.239012055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.071005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zx8\" (UniqueName: \"kubernetes.io/projected/d1351b34-0a6a-499c-95cf-f5c292809054-kube-api-access-d2zx8\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.071038 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.071033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32d5d2b7-74a2-4e98-8344-89d03e643638-data-volume\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.071427 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.071037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps72t\" (UniqueName: \"kubernetes.io/projected/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-kube-api-access-ps72t\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.071427 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.071052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1351b34-0a6a-499c-95cf-f5c292809054-tmp-dir\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.071669 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.071650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1351b34-0a6a-499c-95cf-f5c292809054-config-volume\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.081903 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.081873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zx8\" (UniqueName: \"kubernetes.io/projected/d1351b34-0a6a-499c-95cf-f5c292809054-kube-api-access-d2zx8\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.081998 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.081913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7qw\" (UniqueName: \"kubernetes.io/projected/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-api-access-rv7qw\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.171489 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.171450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps72t\" (UniqueName: \"kubernetes.io/projected/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-kube-api-access-ps72t\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.171666 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.171529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-cert\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.173977 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.173953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-cert\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.179642 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.179615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps72t\" (UniqueName: \"kubernetes.io/projected/a607f5d6-e98b-497c-b6c6-68f68e0bb1e6-kube-api-access-ps72t\") pod \"ingress-canary-skjpn\" (UID: \"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6\") " pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.248852 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.248816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-skjpn"
Sep 17 21:58:52.468314 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.468280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:58:52.468479 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.468285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:58:52.468479 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.468403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:58:52.471245 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.471220    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:52.471384 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.471271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxqmr\""
Sep 17 21:58:52.471384 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.471284    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:52.471512 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.471385    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:52.471512 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.471462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:52.472656 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.472638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4zg8g\""
Sep 17 21:58:52.554028 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.554008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lzzd9_39743950-452b-46ba-95ce-b69a6ea9c790/node-ca/0.log"
Sep 17 21:58:52.569670 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.569645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-skjpn"]
Sep 17 21:58:52.573737 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:52.573715    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda607f5d6_e98b_497c_b6c6_68f68e0bb1e6.slice/crio-e8e0de2b06f17359315cdf31a1b51a050108c4ababd417678bf4823097621efe WatchSource:0}: Error finding container e8e0de2b06f17359315cdf31a1b51a050108c4ababd417678bf4823097621efe: Status 404 returned error can't find the container with id e8e0de2b06f17359315cdf31a1b51a050108c4ababd417678bf4823097621efe
Sep 17 21:58:52.573923 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.573907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.573964 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.573938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:52.573997 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.573987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:52.574051 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574034    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:52.574102 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574088    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.574136 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574104    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.574084364 +0000 UTC m=+38.242147104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:52.574136 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574117    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:52.574214 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574138    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls podName:d1351b34-0a6a-499c-95cf-f5c292809054 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.574123215 +0000 UTC m=+38.242185955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls") pod "dns-default-rrh9f" (UID: "d1351b34-0a6a-499c-95cf-f5c292809054") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.574214 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:52.574157    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.574146267 +0000 UTC m=+38.242209007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:52.620473 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.620395    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="bc461cead2c09e52cf69029d2de11854249b9cc7283ea86615bea57ad8e364c6" exitCode=0
Sep 17 21:58:52.620473 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.620461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"bc461cead2c09e52cf69029d2de11854249b9cc7283ea86615bea57ad8e364c6"}
Sep 17 21:58:52.621683 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.621665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-skjpn" event={"ID":"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6","Type":"ContainerStarted","Data":"e8e0de2b06f17359315cdf31a1b51a050108c4ababd417678bf4823097621efe"}
Sep 17 21:58:52.953016 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.952937    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-42vzf"]
Sep 17 21:58:52.969410 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.969386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-42vzf"]
Sep 17 21:58:52.969533 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.969472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:52.973002 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.972980    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 21:58:52.973150 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.973132    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 21:58:52.973216 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.973132    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jgf4x\""
Sep 17 21:58:52.973262 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.973211    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 21:58:52.973326 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:52.973308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 21:58:53.077927 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.077889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.077927 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.077932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-key\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.078118 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.077954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqh78\" (UniqueName: \"kubernetes.io/projected/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-kube-api-access-gqh78\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.178812 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.178781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.178984 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.178824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-key\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.178984 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.178854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqh78\" (UniqueName: \"kubernetes.io/projected/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-kube-api-access-gqh78\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.179668 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.179643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.182643 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.182615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-signing-key\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.188594 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.188570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqh78\" (UniqueName: \"kubernetes.io/projected/58346833-41f3-4c13-a9ab-a5cb5fb3dc5f-kube-api-access-gqh78\") pod \"service-ca-55f6fc7b86-42vzf\" (UID: \"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.278269 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.278186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf"
Sep 17 21:58:53.564323 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.564289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-42vzf"]
Sep 17 21:58:53.570168 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:58:53.570133    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58346833_41f3_4c13_a9ab_a5cb5fb3dc5f.slice/crio-d2bf8f9170e5c1d24ab3a10a4d0dad8d36a478a4db0b184ef7a3db84cac64788 WatchSource:0}: Error finding container d2bf8f9170e5c1d24ab3a10a4d0dad8d36a478a4db0b184ef7a3db84cac64788: Status 404 returned error can't find the container with id d2bf8f9170e5c1d24ab3a10a4d0dad8d36a478a4db0b184ef7a3db84cac64788
Sep 17 21:58:53.581677 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.581652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:53.581795 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.581715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:53.581795 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.581745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:53.581895 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581819    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:53.581895 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581845    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:53.581895 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581864    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:53.582009 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581897    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls podName:d1351b34-0a6a-499c-95cf-f5c292809054 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.58187504 +0000 UTC m=+40.249937770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls") pod "dns-default-rrh9f" (UID: "d1351b34-0a6a-499c-95cf-f5c292809054") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:53.582009 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581918    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.581907679 +0000 UTC m=+40.249970403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:53.582009 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:53.581933    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.58192492 +0000 UTC m=+40.249987648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:53.628510 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.628328    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b685a8-ec3e-4a39-92f6-deb21d8dba13" containerID="a69207ab8f5649c2c41bfa3a0984efbb29e85c22249f8b1af2b6463364431b17" exitCode=0
Sep 17 21:58:53.628691 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.628413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerDied","Data":"a69207ab8f5649c2c41bfa3a0984efbb29e85c22249f8b1af2b6463364431b17"}
Sep 17 21:58:53.629954 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:53.629930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf" event={"ID":"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f","Type":"ContainerStarted","Data":"d2bf8f9170e5c1d24ab3a10a4d0dad8d36a478a4db0b184ef7a3db84cac64788"}
Sep 17 21:58:54.637174 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:54.637139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" event={"ID":"e1b685a8-ec3e-4a39-92f6-deb21d8dba13","Type":"ContainerStarted","Data":"2556c958fa6d1cdd35b78d97d22c1e568d283abc2e15251eb00ffa2249f8792b"}
Sep 17 21:58:54.663650 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:54.663590    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w2tk2" podStartSLOduration=9.834357083 podStartE2EDuration="37.663573145s" totalImagesPullingTime="27.829216062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:54.663537484 +0000 UTC m=+39.331600256" watchObservedRunningTime="2026-09-17 21:58:54.663573145 +0000 UTC m=+39.331635910"
Sep 17 21:58:55.600945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:55.600907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:55.600945 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:55.600950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:55.601008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601051    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601115    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601128    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:59.601108161 +0000 UTC m=+44.269170899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601130    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601166    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls podName:d1351b34-0a6a-499c-95cf-f5c292809054 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:59.601151331 +0000 UTC m=+44.269214083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls") pod "dns-default-rrh9f" (UID: "d1351b34-0a6a-499c-95cf-f5c292809054") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.601204 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:55.601182    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:59.601174914 +0000 UTC m=+44.269237640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:55.642063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:55.641963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-skjpn" event={"ID":"a607f5d6-e98b-497c-b6c6-68f68e0bb1e6","Type":"ContainerStarted","Data":"ee588eb55e17868f0fb332fd8677297ac7dcd9c9ec300cd5e3e98602d5cbe13e"}
Sep 17 21:58:55.658927 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:55.658877    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-skjpn" podStartSLOduration=2.147067538 podStartE2EDuration="4.658857776s" totalImagesPullingTime="2.511790238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:55.656860864 +0000 UTC m=+40.324923626" watchObservedRunningTime="2026-09-17 21:58:55.658857776 +0000 UTC m=+40.326920525"
Sep 17 21:58:57.648466 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:57.648428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf" event={"ID":"58346833-41f3-4c13-a9ab-a5cb5fb3dc5f","Type":"ContainerStarted","Data":"87bd58e90ea269ac0825780d7f394bdcd1d4871ed699c670a5e3e16affb037b1"}
Sep 17 21:58:57.669398 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:57.669355    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-42vzf" podStartSLOduration=2.367906944 podStartE2EDuration="5.669341399s" totalImagesPullingTime="3.301434455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:57.66884173 +0000 UTC m=+42.336904501" watchObservedRunningTime="2026-09-17 21:58:57.669341399 +0000 UTC m=+42.337404145"
Sep 17 21:58:59.635285 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:59.635248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:59.635307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635370    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635407    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635416    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:07.635404563 +0000 UTC m=+52.303467287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:58:59.635472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls podName:d1351b34-0a6a-499c-95cf-f5c292809054 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:07.635530438 +0000 UTC m=+52.303593162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls") pod "dns-default-rrh9f" (UID: "d1351b34-0a6a-499c-95cf-f5c292809054") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635595    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:59.635745 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:58:59.635656    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:07.635640169 +0000 UTC m=+52.303702908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:07.698987 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.698950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:59:07.699474 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.699016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:07.699474 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.699040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:07.699474 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:59:07.699145    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:07.699474 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:59:07.699211    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls podName:32d5d2b7-74a2-4e98-8344-89d03e643638 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:23.699193813 +0000 UTC m=+68.367256539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkfpl" (UID: "32d5d2b7-74a2-4e98-8344-89d03e643638") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:07.699712 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.699688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32d5d2b7-74a2-4e98-8344-89d03e643638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:07.702067 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.702047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1351b34-0a6a-499c-95cf-f5c292809054-metrics-tls\") pod \"dns-default-rrh9f\" (UID: \"d1351b34-0a6a-499c-95cf-f5c292809054\") " pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:59:07.815681 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:07.815644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:59:08.114064 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:08.114036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrh9f"]
Sep 17 21:59:08.117787 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:08.117751    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1351b34_0a6a_499c_95cf_f5c292809054.slice/crio-8b9865096efd690f5f6df7d9adea7ed20cd6819edb062c02c333b2dbb5083228 WatchSource:0}: Error finding container 8b9865096efd690f5f6df7d9adea7ed20cd6819edb062c02c333b2dbb5083228: Status 404 returned error can't find the container with id 8b9865096efd690f5f6df7d9adea7ed20cd6819edb062c02c333b2dbb5083228
Sep 17 21:59:08.678807 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:08.678770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrh9f" event={"ID":"d1351b34-0a6a-499c-95cf-f5c292809054","Type":"ContainerStarted","Data":"8b9865096efd690f5f6df7d9adea7ed20cd6819edb062c02c333b2dbb5083228"}
Sep 17 21:59:10.684737 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:10.684700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrh9f" event={"ID":"d1351b34-0a6a-499c-95cf-f5c292809054","Type":"ContainerStarted","Data":"ee788c5ca4716a006eff1c750b434a2108c7b15d51665d7044a9a233331cd889"}
Sep 17 21:59:10.685124 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:10.684779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrh9f" event={"ID":"d1351b34-0a6a-499c-95cf-f5c292809054","Type":"ContainerStarted","Data":"521a12d2f1f3422e03ef84925f7a67d6a375c7cf52e45e11c7ae618793615e52"}
Sep 17 21:59:10.685124 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:10.684827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:59:10.702330 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:10.702282    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rrh9f" podStartSLOduration=17.814942572 podStartE2EDuration="19.702268522s" totalImagesPullingTime="1.88732595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:10.701235977 +0000 UTC m=+55.369298745" watchObservedRunningTime="2026-09-17 21:59:10.702268522 +0000 UTC m=+55.370331263"
Sep 17 21:59:12.003276 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.003242    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"]
Sep 17 21:59:12.022615 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.022586    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"]
Sep 17 21:59:12.022764 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.022643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.027304 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.027278    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:12.027304 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.027299    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:12.027522 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.027279    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8945h\""
Sep 17 21:59:12.027522 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.027382    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:12.027911 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.027892    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 21:59:12.040577 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.040541    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"]
Sep 17 21:59:12.040577 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.040584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"]
Sep 17 21:59:12.040769 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.040680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.045405 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.045386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 21:59:12.045894 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.045875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 21:59:12.045894 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.045898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 21:59:12.046250 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.046227    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 21:59:12.134695 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.134875 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkf8\" (UniqueName: \"kubernetes.io/projected/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-kube-api-access-pkkf8\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.134875 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.134875 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.134875 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtgt\" (UniqueName: \"kubernetes.io/projected/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-kube-api-access-vvtgt\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.135037 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.135037 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.135037 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.134985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.235615 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.235767 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtgt\" (UniqueName: \"kubernetes.io/projected/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-kube-api-access-vvtgt\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.235767 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.235767 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.235767 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.236001 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.236001 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkf8\" (UniqueName: \"kubernetes.io/projected/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-kube-api-access-pkkf8\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.236001 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.235860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.236907 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.236873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.238604 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.238583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.238694 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.238614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.238694 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.238663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.238694 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.238680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.239023 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.239002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-ca\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.244963 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.244938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkf8\" (UniqueName: \"kubernetes.io/projected/19ac1ef7-7b92-41bd-9980-b0bc622f2c65-kube-api-access-pkkf8\") pod \"cluster-proxy-proxy-agent-cc9dcdd78-c65n2\" (UID: \"19ac1ef7-7b92-41bd-9980-b0bc622f2c65\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.245165 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.245146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtgt\" (UniqueName: \"kubernetes.io/projected/b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c-kube-api-access-vvtgt\") pod \"managed-serviceaccount-addon-agent-54bbbc75cb-c52bl\" (UID: \"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.342966 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.342931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"
Sep 17 21:59:12.350133 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.350098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"
Sep 17 21:59:12.641369 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.641267    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl"]
Sep 17 21:59:12.645111 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:12.645086    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb103e3fd_7b5a_43cd_9dc2_9a2a22d88f3c.slice/crio-17e58588560dcec42098a1a1395d94c1695470ddedd19785f8b6b84a2440d04b WatchSource:0}: Error finding container 17e58588560dcec42098a1a1395d94c1695470ddedd19785f8b6b84a2440d04b: Status 404 returned error can't find the container with id 17e58588560dcec42098a1a1395d94c1695470ddedd19785f8b6b84a2440d04b
Sep 17 21:59:12.669529 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.669501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2"]
Sep 17 21:59:12.672996 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:12.672958    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19ac1ef7_7b92_41bd_9980_b0bc622f2c65.slice/crio-24ade10596bf70af06c5d83426ac6241729a36b182d1014489923e016a993305 WatchSource:0}: Error finding container 24ade10596bf70af06c5d83426ac6241729a36b182d1014489923e016a993305: Status 404 returned error can't find the container with id 24ade10596bf70af06c5d83426ac6241729a36b182d1014489923e016a993305
Sep 17 21:59:12.690527 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.690499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2" event={"ID":"19ac1ef7-7b92-41bd-9980-b0bc622f2c65","Type":"ContainerStarted","Data":"24ade10596bf70af06c5d83426ac6241729a36b182d1014489923e016a993305"}
Sep 17 21:59:12.691572 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:12.691535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl" event={"ID":"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c","Type":"ContainerStarted","Data":"17e58588560dcec42098a1a1395d94c1695470ddedd19785f8b6b84a2440d04b"}
Sep 17 21:59:16.621150 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:16.621123    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pjh7"
Sep 17 21:59:16.705844 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:16.705761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl" event={"ID":"b103e3fd-7b5a-43cd-9dc2-9a2a22d88f3c","Type":"ContainerStarted","Data":"f6f0165e9422cdcb08acd735467554046fe213297b482171f1f8bd1ba3041020"}
Sep 17 21:59:16.723052 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:16.722994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bbbc75cb-c52bl" podStartSLOduration=1.923566516 podStartE2EDuration="5.72297754s" totalImagesPullingTime="3.799411024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:16.722374073 +0000 UTC m=+61.390436899" watchObservedRunningTime="2026-09-17 21:59:16.72297754 +0000 UTC m=+61.391040287"
Sep 17 21:59:19.717192 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:19.717154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2" event={"ID":"19ac1ef7-7b92-41bd-9980-b0bc622f2c65","Type":"ContainerStarted","Data":"d02aa9188fe77fb7a6bc6414f8a770bd0d8a8d15b21a6cdaaa07e4f2d16bd443"}
Sep 17 21:59:19.717192 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:19.717198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2" event={"ID":"19ac1ef7-7b92-41bd-9980-b0bc622f2c65","Type":"ContainerStarted","Data":"366f3486e52cf8b2cfee8c24f93b116dd6d2fa80317580d9b33d77c3c6ac3fb0"}
Sep 17 21:59:19.717662 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:19.717210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2" event={"ID":"19ac1ef7-7b92-41bd-9980-b0bc622f2c65","Type":"ContainerStarted","Data":"fe1219a800b957846e9784d7bdec0a455bd9d66bdda7a8980742882d84b08e9a"}
Sep 17 21:59:19.740366 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:19.740313    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc9dcdd78-c65n2" podStartSLOduration=2.230471758 podStartE2EDuration="8.740296907s" totalImagesPullingTime="6.509825149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:19.739741844 +0000 UTC m=+64.407804592" watchObservedRunningTime="2026-09-17 21:59:19.740296907 +0000 UTC m=+64.408359657"
Sep 17 21:59:20.689514 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:20.689478    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rrh9f"
Sep 17 21:59:23.124124 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.124095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:59:23.126949 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.126932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:23.137350 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.137320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbff581-3aee-4f8c-89e8-e47b010af5fd-metrics-certs\") pod \"network-metrics-daemon-24p4h\" (UID: \"cfbff581-3aee-4f8c-89e8-e47b010af5fd\") " pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:59:23.224632 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.224591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:59:23.224816 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.224648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:59:23.227484 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.227461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:23.227635 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.227610    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:23.237505 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.237487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3397b63b-b850-4a86-9411-754f199657fe-original-pull-secret\") pod \"global-pull-secret-syncer-st4hm\" (UID: \"3397b63b-b850-4a86-9411-754f199657fe\") " pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:59:23.237938 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.237926    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:23.248392 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.248371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcqk\" (UniqueName: \"kubernetes.io/projected/4819ba77-53a1-4a53-9643-699d42acdff5-kube-api-access-dbcqk\") pod \"network-check-target-jkjz9\" (UID: \"4819ba77-53a1-4a53-9643-699d42acdff5\") " pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:59:23.381376 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.381229    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4zg8g\""
Sep 17 21:59:23.386141 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.386121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxqmr\""
Sep 17 21:59:23.388245 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.388228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-st4hm"
Sep 17 21:59:23.389398 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.389383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:59:23.394050 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.394029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24p4h"
Sep 17 21:59:23.666966 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.666886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jkjz9"]
Sep 17 21:59:23.669659 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.669636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-st4hm"]
Sep 17 21:59:23.670694 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:23.670670    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4819ba77_53a1_4a53_9643_699d42acdff5.slice/crio-377e67f90146cfcd0d37df16b707984cb448ce5fa5fb042e2c72ced09d045bae WatchSource:0}: Error finding container 377e67f90146cfcd0d37df16b707984cb448ce5fa5fb042e2c72ced09d045bae: Status 404 returned error can't find the container with id 377e67f90146cfcd0d37df16b707984cb448ce5fa5fb042e2c72ced09d045bae
Sep 17 21:59:23.672006 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:23.671988    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3397b63b_b850_4a86_9411_754f199657fe.slice/crio-dd89fa70ddc2725df553cb60256af0998079fc1b13994e19af3fc2ade45af13f WatchSource:0}: Error finding container dd89fa70ddc2725df553cb60256af0998079fc1b13994e19af3fc2ade45af13f: Status 404 returned error can't find the container with id dd89fa70ddc2725df553cb60256af0998079fc1b13994e19af3fc2ade45af13f
Sep 17 21:59:23.677146 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.677125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24p4h"]
Sep 17 21:59:23.682068 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:23.682046    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbff581_3aee_4f8c_89e8_e47b010af5fd.slice/crio-348f769176897ca8c4967b9a3d98bf3630eadace5a155f812e3484c32339f12a WatchSource:0}: Error finding container 348f769176897ca8c4967b9a3d98bf3630eadace5a155f812e3484c32339f12a: Status 404 returned error can't find the container with id 348f769176897ca8c4967b9a3d98bf3630eadace5a155f812e3484c32339f12a
Sep 17 21:59:23.728473 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.728445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:23.730119 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.730094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jkjz9" event={"ID":"4819ba77-53a1-4a53-9643-699d42acdff5","Type":"ContainerStarted","Data":"377e67f90146cfcd0d37df16b707984cb448ce5fa5fb042e2c72ced09d045bae"}
Sep 17 21:59:23.731064 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.731040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32d5d2b7-74a2-4e98-8344-89d03e643638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkfpl\" (UID: \"32d5d2b7-74a2-4e98-8344-89d03e643638\") " pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:23.731293 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.731271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24p4h" event={"ID":"cfbff581-3aee-4f8c-89e8-e47b010af5fd","Type":"ContainerStarted","Data":"348f769176897ca8c4967b9a3d98bf3630eadace5a155f812e3484c32339f12a"}
Sep 17 21:59:23.732282 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.732265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-st4hm" event={"ID":"3397b63b-b850-4a86-9411-754f199657fe","Type":"ContainerStarted","Data":"dd89fa70ddc2725df553cb60256af0998079fc1b13994e19af3fc2ade45af13f"}
Sep 17 21:59:23.742277 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.742260    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qxfkf\""
Sep 17 21:59:23.749777 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:23.749759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkfpl"
Sep 17 21:59:24.020442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:24.020359    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkfpl"]
Sep 17 21:59:24.023869 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:24.023843    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d5d2b7_74a2_4e98_8344_89d03e643638.slice/crio-796ca20136861babf5a0a51b1ddb480d4b3cc38caf4f8ac63fbfa7f4eb4601b5 WatchSource:0}: Error finding container 796ca20136861babf5a0a51b1ddb480d4b3cc38caf4f8ac63fbfa7f4eb4601b5: Status 404 returned error can't find the container with id 796ca20136861babf5a0a51b1ddb480d4b3cc38caf4f8ac63fbfa7f4eb4601b5
Sep 17 21:59:24.739265 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:24.739083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkfpl" event={"ID":"32d5d2b7-74a2-4e98-8344-89d03e643638","Type":"ContainerStarted","Data":"4f9a556294ebb268800a39dec9e55335214045b4f584c0fa6de9b594f9054b46"}
Sep 17 21:59:24.739265 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:24.739196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkfpl" event={"ID":"32d5d2b7-74a2-4e98-8344-89d03e643638","Type":"ContainerStarted","Data":"796ca20136861babf5a0a51b1ddb480d4b3cc38caf4f8ac63fbfa7f4eb4601b5"}
Sep 17 21:59:25.748616 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:25.748448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24p4h" event={"ID":"cfbff581-3aee-4f8c-89e8-e47b010af5fd","Type":"ContainerStarted","Data":"9a6715dcc7b34fa179ddd0a4216eef77dae7bfdd61ecae6bf2d8b57b4c9dda1f"}
Sep 17 21:59:25.748616 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:25.748517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24p4h" event={"ID":"cfbff581-3aee-4f8c-89e8-e47b010af5fd","Type":"ContainerStarted","Data":"5585e2d9a0f49baa6029c876e07ebb39e440f85de2eba51d1d947954e806aa62"}
Sep 17 21:59:25.751776 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:25.751705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkfpl" event={"ID":"32d5d2b7-74a2-4e98-8344-89d03e643638","Type":"ContainerStarted","Data":"afd24520d13ae3a80c6ccc0600267af0f84db953a2aad848123c15c69fe41235"}
Sep 17 21:59:25.770986 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:25.770932    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-24p4h" podStartSLOduration=67.761971971 podStartE2EDuration="1m8.77091663s" totalImagesPullingTime="1.008944659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:25.769492128 +0000 UTC m=+70.437554895" watchObservedRunningTime="2026-09-17 21:59:25.77091663 +0000 UTC m=+70.438979420"
Sep 17 21:59:28.766470 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.764752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jkjz9" event={"ID":"4819ba77-53a1-4a53-9643-699d42acdff5","Type":"ContainerStarted","Data":"692f5eb076fa22ac0dc36960b4c7740c5fc30d8f165cac036f511abdb0714936"}
Sep 17 21:59:28.766470 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.765653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 21:59:28.767801 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.767774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-st4hm" event={"ID":"3397b63b-b850-4a86-9411-754f199657fe","Type":"ContainerStarted","Data":"b0bfd3695673da473bd6c955bb58114bd3bf47441eec855c77ca02fa0cef6771"}
Sep 17 21:59:28.770071 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.770048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkfpl" event={"ID":"32d5d2b7-74a2-4e98-8344-89d03e643638","Type":"ContainerStarted","Data":"916f72dc57b51c04128cabd9dde7c0eeb17b1efc94adcbe9a08bb68aa9804706"}
Sep 17 21:59:28.783136 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.783081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jkjz9" podStartSLOduration=66.884826208 podStartE2EDuration="1m11.783062109s" totalImagesPullingTime="4.898235901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:28.782377905 +0000 UTC m=+73.450440675" watchObservedRunningTime="2026-09-17 21:59:28.783062109 +0000 UTC m=+73.451124857"
Sep 17 21:59:28.798231 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.798183    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-st4hm" podStartSLOduration=66.889411652 podStartE2EDuration="1m11.798168838s" totalImagesPullingTime="4.908757186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:28.798066195 +0000 UTC m=+73.466128944" watchObservedRunningTime="2026-09-17 21:59:28.798168838 +0000 UTC m=+73.466231584"
Sep 17 21:59:28.817978 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:28.817879    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkfpl" podStartSLOduration=33.416112471 podStartE2EDuration="37.817861698s" totalImagesPullingTime="4.401749227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:28.817312162 +0000 UTC m=+73.485374939" watchObservedRunningTime="2026-09-17 21:59:28.817861698 +0000 UTC m=+73.485924469"
Sep 17 21:59:31.736265 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.736228    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pk7hm"]
Sep 17 21:59:31.738294 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.738274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.742109 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.742085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:59:31.742109 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.742089    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:59:31.743144 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.743127    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:59:31.743144 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.743139    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:59:31.743380 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.743366    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:31.743475 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.743458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5x5d5\""
Sep 17 21:59:31.743797 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.743541    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:31.781948 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.781912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcvh\" (UniqueName: \"kubernetes.io/projected/dee56a9e-9adc-44aa-bea2-5e22784fb30d-kube-api-access-vzcvh\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782116 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.781955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782116 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.781984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-root\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782116 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-sys\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782116 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-metrics-client-ca\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782116 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782344 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-wtmp\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782344 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.782344 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.782241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-textfile\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-sys\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883063 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-metrics-client-ca\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-wtmp\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-sys\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-textfile\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcvh\" (UniqueName: \"kubernetes.io/projected/dee56a9e-9adc-44aa-bea2-5e22784fb30d-kube-api-access-vzcvh\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883318 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-root\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883735 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-root\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883735 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-textfile\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883735 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-wtmp\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883881 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:59:31.883756    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 21:59:31.883881 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-metrics-client-ca\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.883881 ip-10-0-136-9 kubenswrapper[2582]: E0917 21:59:31.883810    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls podName:dee56a9e-9adc-44aa-bea2-5e22784fb30d nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.383792559 +0000 UTC m=+77.051855289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls") pod "node-exporter-pk7hm" (UID: "dee56a9e-9adc-44aa-bea2-5e22784fb30d") : secret "node-exporter-tls" not found
Sep 17 21:59:31.883881 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.883838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.885971 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.885948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:31.891905 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:31.891886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcvh\" (UniqueName: \"kubernetes.io/projected/dee56a9e-9adc-44aa-bea2-5e22784fb30d-kube-api-access-vzcvh\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:32.387903 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:32.387862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:32.391155 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:32.391123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dee56a9e-9adc-44aa-bea2-5e22784fb30d-node-exporter-tls\") pod \"node-exporter-pk7hm\" (UID: \"dee56a9e-9adc-44aa-bea2-5e22784fb30d\") " pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:32.650304 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:32.650216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pk7hm"
Sep 17 21:59:32.660273 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:32.660246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee56a9e_9adc_44aa_bea2_5e22784fb30d.slice/crio-633579d2048cd87c5bfc860f80ce121c6198b622a6a731ea3b1df1e759f8f4d8 WatchSource:0}: Error finding container 633579d2048cd87c5bfc860f80ce121c6198b622a6a731ea3b1df1e759f8f4d8: Status 404 returned error can't find the container with id 633579d2048cd87c5bfc860f80ce121c6198b622a6a731ea3b1df1e759f8f4d8
Sep 17 21:59:32.784975 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:32.784940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pk7hm" event={"ID":"dee56a9e-9adc-44aa-bea2-5e22784fb30d","Type":"ContainerStarted","Data":"633579d2048cd87c5bfc860f80ce121c6198b622a6a731ea3b1df1e759f8f4d8"}
Sep 17 21:59:33.795335 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:33.795240    2582 generic.go:356] "Generic (PLEG): container finished" podID="dee56a9e-9adc-44aa-bea2-5e22784fb30d" containerID="79befaf2ac0b2041cb4db73ed10238c85e5eae004b4ff29a956665e58a4ee680" exitCode=0
Sep 17 21:59:33.795335 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:33.795297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pk7hm" event={"ID":"dee56a9e-9adc-44aa-bea2-5e22784fb30d","Type":"ContainerDied","Data":"79befaf2ac0b2041cb4db73ed10238c85e5eae004b4ff29a956665e58a4ee680"}
Sep 17 21:59:34.802301 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:34.802270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pk7hm" event={"ID":"dee56a9e-9adc-44aa-bea2-5e22784fb30d","Type":"ContainerStarted","Data":"005a71571b6ad6ba65650a1966e91cde491b16f9bd174dcd67261e04f113518e"}
Sep 17 21:59:34.802712 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:34.802320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pk7hm" event={"ID":"dee56a9e-9adc-44aa-bea2-5e22784fb30d","Type":"ContainerStarted","Data":"0afb7e1c5946b26334bf893905451308fd6af64cf416e1cb14e332048edf61c6"}
Sep 17 21:59:34.824500 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:34.824454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pk7hm" podStartSLOduration=2.918568226 podStartE2EDuration="3.824439361s" totalImagesPullingTime="843.073676ms" totalInitContainerRuntime="62.797459ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.823004111 +0000 UTC m=+79.491066910" watchObservedRunningTime="2026-09-17 21:59:34.824439361 +0000 UTC m=+79.492502148"
Sep 17 21:59:36.437091 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.437057    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"]
Sep 17 21:59:36.438962 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.438944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.441378 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.441352    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 21:59:36.441536 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.441517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v4ptc\""
Sep 17 21:59:36.450159 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.450136    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"]
Sep 17 21:59:36.517438 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.517403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lhbb\" (UniqueName: \"kubernetes.io/projected/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-kube-api-access-6lhbb\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.517438 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.517435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.617725 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.617695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lhbb\" (UniqueName: \"kubernetes.io/projected/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-kube-api-access-6lhbb\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.617725 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.617729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.620172 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.620150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.627989 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.627965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lhbb\" (UniqueName: \"kubernetes.io/projected/0ea5405c-b3bf-4b09-96b8-66292f9d03f5-kube-api-access-6lhbb\") pod \"monitoring-plugin-5984459ff8-jlr7j\" (UID: \"0ea5405c-b3bf-4b09-96b8-66292f9d03f5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:36.749805 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:36.749718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:37.021887 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:37.021803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"]
Sep 17 21:59:37.025021 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:37.024990    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea5405c_b3bf_4b09_96b8_66292f9d03f5.slice/crio-8bb3205ed0de04e1885cfdd9e7e44695a3e46a773bb87dc393a01a6af9cd90cc WatchSource:0}: Error finding container 8bb3205ed0de04e1885cfdd9e7e44695a3e46a773bb87dc393a01a6af9cd90cc: Status 404 returned error can't find the container with id 8bb3205ed0de04e1885cfdd9e7e44695a3e46a773bb87dc393a01a6af9cd90cc
Sep 17 21:59:37.814181 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:37.814123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j" event={"ID":"0ea5405c-b3bf-4b09-96b8-66292f9d03f5","Type":"ContainerStarted","Data":"8bb3205ed0de04e1885cfdd9e7e44695a3e46a773bb87dc393a01a6af9cd90cc"}
Sep 17 21:59:38.819493 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:38.819459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j" event={"ID":"0ea5405c-b3bf-4b09-96b8-66292f9d03f5","Type":"ContainerStarted","Data":"37aa6171670630b2226a634bc0ecb84c903d7c85c5b1d9f6bdf4c6373af04747"}
Sep 17 21:59:38.819912 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:38.819647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:38.824592 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:38.824570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j"
Sep 17 21:59:38.835869 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:38.835813    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jlr7j" podStartSLOduration=1.646786572 podStartE2EDuration="2.835796549s" totalImagesPullingTime="1.189009977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:38.834795874 +0000 UTC m=+83.502858620" watchObservedRunningTime="2026-09-17 21:59:38.835796549 +0000 UTC m=+83.503859293"
Sep 17 21:59:44.928144 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.928109    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 21:59:44.930814 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.930796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.933727 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.933705    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 21:59:44.933916 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.933708    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 21:59:44.933916 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.933739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 21:59:44.934977 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.934959    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 21:59:44.935119 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.935014    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 21:59:44.935119 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.935035    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 21:59:44.935221 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.935208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rhsvk\""
Sep 17 21:59:44.935330 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.935308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 21:59:44.939965 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.939937    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 21:59:44.940785 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.940762    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 21:59:44.977802 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.977958 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.977958 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.977958 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.977958 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.977958 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:44.978119 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:44.977968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn7w\" (UniqueName: \"kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079245 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079402 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079402 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn7w\" (UniqueName: \"kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079402 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079402 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079402 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.079664 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.079471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.080090 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.080059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.080205 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.080115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.080205 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.080193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.080413 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.080396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.081889 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.081870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.081949 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.081885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.087679 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.087657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn7w\" (UniqueName: \"kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w\") pod \"console-5fd4f7d779-t7gjb\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") " pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.240442 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.240352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:45.521651 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.521569    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 21:59:45.524817 ip-10-0-136-9 kubenswrapper[2582]: W0917 21:59:45.524794    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d843fe_d20d_4953_a795_857da3bc0e20.slice/crio-5eb272deaee24b2506024e1347cbd7f979325feb8d31653c83f518ac7b7e7bf6 WatchSource:0}: Error finding container 5eb272deaee24b2506024e1347cbd7f979325feb8d31653c83f518ac7b7e7bf6: Status 404 returned error can't find the container with id 5eb272deaee24b2506024e1347cbd7f979325feb8d31653c83f518ac7b7e7bf6
Sep 17 21:59:45.846233 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:45.846171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4f7d779-t7gjb" event={"ID":"d8d843fe-d20d-4953-a795-857da3bc0e20","Type":"ContainerStarted","Data":"5eb272deaee24b2506024e1347cbd7f979325feb8d31653c83f518ac7b7e7bf6"}
Sep 17 21:59:49.863609 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:49.863570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4f7d779-t7gjb" event={"ID":"d8d843fe-d20d-4953-a795-857da3bc0e20","Type":"ContainerStarted","Data":"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"}
Sep 17 21:59:50.888365 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:50.888313    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fd4f7d779-t7gjb" podStartSLOduration=2.665412467 podStartE2EDuration="6.8882983s" totalImagesPullingTime="4.222885833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.887092573 +0000 UTC m=+95.555155320" watchObservedRunningTime="2026-09-17 21:59:50.8882983 +0000 UTC m=+95.556361046"
Sep 17 21:59:55.240789 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:55.240741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:55.240789 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:55.240805    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:55.245490 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:55.245465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 21:59:55.887668 ip-10-0-136-9 kubenswrapper[2582]: I0917 21:59:55.887642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 22:00:00.778891 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:00.778859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jkjz9"
Sep 17 22:00:59.593527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.593297    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:00:59.596124 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.596096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.607228 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.607202    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:00:59.714687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qr6\" (UniqueName: \"kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714986 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.714986 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.714884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816177 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816177 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qr6\" (UniqueName: \"kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.816403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.816391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.817118 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.817092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.817226 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.817117    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.817226 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.817117    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.817508 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.817487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.818709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.818688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.818881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.818867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.825065 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.825046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qr6\" (UniqueName: \"kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6\") pod \"console-77ccb554b4-qt2sm\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") " pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:00:59.904849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:00:59.904760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:01:00.173981 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:00.173911    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:01:00.177010 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:01:00.176983    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72039752_7d82_484c_8509_8d671b67f896.slice/crio-dfe022e0dad03890600dc2a158e3de0c1d0500c47660c2e429c7ce3550799d06 WatchSource:0}: Error finding container dfe022e0dad03890600dc2a158e3de0c1d0500c47660c2e429c7ce3550799d06: Status 404 returned error can't find the container with id dfe022e0dad03890600dc2a158e3de0c1d0500c47660c2e429c7ce3550799d06
Sep 17 22:01:01.100532 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:01.100491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ccb554b4-qt2sm" event={"ID":"72039752-7d82-484c-8509-8d671b67f896","Type":"ContainerStarted","Data":"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"}
Sep 17 22:01:01.100532 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:01.100541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ccb554b4-qt2sm" event={"ID":"72039752-7d82-484c-8509-8d671b67f896","Type":"ContainerStarted","Data":"dfe022e0dad03890600dc2a158e3de0c1d0500c47660c2e429c7ce3550799d06"}
Sep 17 22:01:01.118625 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:01.118579    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77ccb554b4-qt2sm" podStartSLOduration=2.118561901 podStartE2EDuration="2.118561901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:01:01.116808504 +0000 UTC m=+165.784871272" watchObservedRunningTime="2026-09-17 22:01:01.118561901 +0000 UTC m=+165.786624639"
Sep 17 22:01:09.904930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:09.904872    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:01:09.905385 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:09.904973    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:01:09.909956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:09.909930    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:01:10.136402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:10.136368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:01:10.183032 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:10.182962    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 22:01:35.202131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.202058    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fd4f7d779-t7gjb" podUID="d8d843fe-d20d-4953-a795-857da3bc0e20" containerName="console" containerID="cri-o://d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144" gracePeriod=15
Sep 17 22:01:35.448971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.448945    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 22:01:35.458889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.458889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.458889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.459082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.459082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.459082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.458985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.459082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.459015    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdn7w\" (UniqueName: \"kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w\") pod \"d8d843fe-d20d-4953-a795-857da3bc0e20\" (UID: \"d8d843fe-d20d-4953-a795-857da3bc0e20\") "
Sep 17 22:01:35.459271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.459240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config" (OuterVolumeSpecName: "console-config") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:35.459328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.459297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:35.459652 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.459538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:35.459745 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.459697    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:01:35.461317 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.461284    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:01:35.461418 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.461328    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:01:35.461418 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.461360    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w" (OuterVolumeSpecName: "kube-api-access-qdn7w") pod "d8d843fe-d20d-4953-a795-857da3bc0e20" (UID: "d8d843fe-d20d-4953-a795-857da3bc0e20"). InnerVolumeSpecName "kube-api-access-qdn7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:01:35.559912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559875    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-console-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.559912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559906    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-oauth-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.559912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559915    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-service-ca\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.559912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559923    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-oauth-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.560174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559931    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8d843fe-d20d-4953-a795-857da3bc0e20-console-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.560174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559940    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8d843fe-d20d-4953-a795-857da3bc0e20-trusted-ca-bundle\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:35.560174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:35.559947    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdn7w\" (UniqueName: \"kubernetes.io/projected/d8d843fe-d20d-4953-a795-857da3bc0e20-kube-api-access-qdn7w\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:01:36.221099 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.221065    2582 generic.go:356] "Generic (PLEG): container finished" podID="d8d843fe-d20d-4953-a795-857da3bc0e20" containerID="d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144" exitCode=0
Sep 17 22:01:36.221500 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.221115    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4f7d779-t7gjb"
Sep 17 22:01:36.221500 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.221138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4f7d779-t7gjb" event={"ID":"d8d843fe-d20d-4953-a795-857da3bc0e20","Type":"ContainerDied","Data":"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"}
Sep 17 22:01:36.221500 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.221187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4f7d779-t7gjb" event={"ID":"d8d843fe-d20d-4953-a795-857da3bc0e20","Type":"ContainerDied","Data":"5eb272deaee24b2506024e1347cbd7f979325feb8d31653c83f518ac7b7e7bf6"}
Sep 17 22:01:36.221500 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.221215    2582 scope.go:117] "RemoveContainer" containerID="d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"
Sep 17 22:01:36.229097 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.229072    2582 scope.go:117] "RemoveContainer" containerID="d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"
Sep 17 22:01:36.229383 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:01:36.229361    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144\": container with ID starting with d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144 not found: ID does not exist" containerID="d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"
Sep 17 22:01:36.229460 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.229391    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144"} err="failed to get container status \"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144\": rpc error: code = NotFound desc = could not find container \"d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144\": container with ID starting with d259ab25799d901470b5c6e72ec11da0f3b658a98af50eefb0b7f4180c478144 not found: ID does not exist"
Sep 17 22:01:36.238446 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.238422    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 22:01:36.240263 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:36.240236    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fd4f7d779-t7gjb"]
Sep 17 22:01:37.472385 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:01:37.472346    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d843fe-d20d-4953-a795-857da3bc0e20" path="/var/lib/kubelet/pods/d8d843fe-d20d-4953-a795-857da3bc0e20/volumes"
Sep 17 22:02:22.084705 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.084668    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:02:22.085086 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.084898    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d843fe-d20d-4953-a795-857da3bc0e20" containerName="console"
Sep 17 22:02:22.085086 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.084909    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d843fe-d20d-4953-a795-857da3bc0e20" containerName="console"
Sep 17 22:02:22.085086 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.084950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d843fe-d20d-4953-a795-857da3bc0e20" containerName="console"
Sep 17 22:02:22.088925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.088908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.098101 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.098073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:02:22.178037 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.177991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178037 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnkf\" (UniqueName: \"kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.178241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.178206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278717 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278717 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278885 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnkf\" (UniqueName: \"kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278885 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278885 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.278885 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.278805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.279318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.280081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.280396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.280378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.280882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.284125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.282242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.286479 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.286456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.288460 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.288438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnkf\" (UniqueName: \"kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf\") pod \"console-86d6d76ccc-wwwdl\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") " pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.397972 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.397881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:22.668944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:22.668864    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:02:22.671808 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:02:22.671770    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a97bec_0b9e_416f_a02c_ebe55b22ef08.slice/crio-3eec0f40280cacc610b19188ad1f6fa9dfd0e3581c8e94a950aa02ff6e413794 WatchSource:0}: Error finding container 3eec0f40280cacc610b19188ad1f6fa9dfd0e3581c8e94a950aa02ff6e413794: Status 404 returned error can't find the container with id 3eec0f40280cacc610b19188ad1f6fa9dfd0e3581c8e94a950aa02ff6e413794
Sep 17 22:02:23.378879 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:23.378844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d6d76ccc-wwwdl" event={"ID":"e6a97bec-0b9e-416f-a02c-ebe55b22ef08","Type":"ContainerStarted","Data":"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"}
Sep 17 22:02:23.379268 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:23.378900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d6d76ccc-wwwdl" event={"ID":"e6a97bec-0b9e-416f-a02c-ebe55b22ef08","Type":"ContainerStarted","Data":"3eec0f40280cacc610b19188ad1f6fa9dfd0e3581c8e94a950aa02ff6e413794"}
Sep 17 22:02:23.399909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:23.399857    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86d6d76ccc-wwwdl" podStartSLOduration=1.399842132 podStartE2EDuration="1.399842132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:02:23.398371505 +0000 UTC m=+248.066434252" watchObservedRunningTime="2026-09-17 22:02:23.399842132 +0000 UTC m=+248.067904880"
Sep 17 22:02:32.398658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:32.398565    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:32.398658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:32.398628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:32.403272 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:32.403252    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:32.413213 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:32.413188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:02:32.471673 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:32.471638    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:02:57.490791 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.490734    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77ccb554b4-qt2sm" podUID="72039752-7d82-484c-8509-8d671b67f896" containerName="console" containerID="cri-o://310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7" gracePeriod=15
Sep 17 22:02:57.720903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.720878    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:02:57.846290 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846240    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846290 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85qr6\" (UniqueName: \"kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846579 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846579 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846579 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846579 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846790 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle\") pod \"72039752-7d82-484c-8509-8d671b67f896\" (UID: \"72039752-7d82-484c-8509-8d671b67f896\") "
Sep 17 22:02:57.846868 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca" (OuterVolumeSpecName: "service-ca") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:57.846868 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846849    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config" (OuterVolumeSpecName: "console-config") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:57.846982 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.846857    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:57.847129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.847107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:57.848575 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.848528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:57.848690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.848572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:57.848690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.848578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6" (OuterVolumeSpecName: "kube-api-access-85qr6") pod "72039752-7d82-484c-8509-8d671b67f896" (UID: "72039752-7d82-484c-8509-8d671b67f896"). InnerVolumeSpecName "kube-api-access-85qr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:57.947687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947650    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qr6\" (UniqueName: \"kubernetes.io/projected/72039752-7d82-484c-8509-8d671b67f896-kube-api-access-85qr6\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947682    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-service-ca\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947691    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-console-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947699    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-oauth-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947709    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-oauth-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947718    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72039752-7d82-484c-8509-8d671b67f896-trusted-ca-bundle\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:57.947917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:57.947726    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72039752-7d82-484c-8509-8d671b67f896-console-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:02:58.494468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.494430    2582 generic.go:356] "Generic (PLEG): container finished" podID="72039752-7d82-484c-8509-8d671b67f896" containerID="310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7" exitCode=0
Sep 17 22:02:58.494468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.494472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ccb554b4-qt2sm"
Sep 17 22:02:58.494914 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.494517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ccb554b4-qt2sm" event={"ID":"72039752-7d82-484c-8509-8d671b67f896","Type":"ContainerDied","Data":"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"}
Sep 17 22:02:58.494914 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.494585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ccb554b4-qt2sm" event={"ID":"72039752-7d82-484c-8509-8d671b67f896","Type":"ContainerDied","Data":"dfe022e0dad03890600dc2a158e3de0c1d0500c47660c2e429c7ce3550799d06"}
Sep 17 22:02:58.494914 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.494606    2582 scope.go:117] "RemoveContainer" containerID="310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"
Sep 17 22:02:58.503311 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.503294    2582 scope.go:117] "RemoveContainer" containerID="310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"
Sep 17 22:02:58.503648 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:02:58.503626    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7\": container with ID starting with 310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7 not found: ID does not exist" containerID="310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"
Sep 17 22:02:58.503722 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.503662    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7"} err="failed to get container status \"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7\": rpc error: code = NotFound desc = could not find container \"310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7\": container with ID starting with 310016d92cbb012fea9003989c285aa7bd51abb2f398c66b752bffa0a4b6cfd7 not found: ID does not exist"
Sep 17 22:02:58.515505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.515477    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:02:58.517956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:58.517933    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77ccb554b4-qt2sm"]
Sep 17 22:02:59.472341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:02:59.472307    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72039752-7d82-484c-8509-8d671b67f896" path="/var/lib/kubelet/pods/72039752-7d82-484c-8509-8d671b67f896/volumes"
Sep 17 22:03:17.331905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:17.331879    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:03:34.975967 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.975932    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-tflrx"]
Sep 17 22:03:34.978245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.976161    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72039752-7d82-484c-8509-8d671b67f896" containerName="console"
Sep 17 22:03:34.978245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.976171    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="72039752-7d82-484c-8509-8d671b67f896" containerName="console"
Sep 17 22:03:34.978245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.976220    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="72039752-7d82-484c-8509-8d671b67f896" containerName="console"
Sep 17 22:03:34.979103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.979088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:34.982150 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.982129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 22:03:34.982580 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.982539    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 22:03:34.983508 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.983490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 22:03:34.983605 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.983515    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nkhlx\""
Sep 17 22:03:34.983605 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.983561    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 22:03:34.988622 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:34.988596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-tflrx"]
Sep 17 22:03:35.085324 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.085292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.085520 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.085334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvc4\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-kube-api-access-kcvc4\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.186660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.186623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.186660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.186663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvc4\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-kube-api-access-kcvc4\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.186884 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:03:35.186800    2582 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:35.186884 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:03:35.186831    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-tflrx: secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:35.186957 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:03:35.186898    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates podName:167219ff-770c-4269-a766-000d8def4b34 nodeName:}" failed. No retries permitted until 2026-09-17 22:03:35.686877007 +0000 UTC m=+320.354939733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates") pod "keda-admission-6c4d864b86-tflrx" (UID: "167219ff-770c-4269-a766-000d8def4b34") : secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:35.195315 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.195288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvc4\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-kube-api-access-kcvc4\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.690528 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.690492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.692892 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.692874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/167219ff-770c-4269-a766-000d8def4b34-certificates\") pod \"keda-admission-6c4d864b86-tflrx\" (UID: \"167219ff-770c-4269-a766-000d8def4b34\") " pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:35.890160 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:35.890127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:36.159292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:36.159260    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-tflrx"]
Sep 17 22:03:36.162858 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:03:36.162825    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167219ff_770c_4269_a766_000d8def4b34.slice/crio-ccdea4af36ba3764fe9ab4be71941d84ff1bf8a4b1b954e9a247e58a15387fe2 WatchSource:0}: Error finding container ccdea4af36ba3764fe9ab4be71941d84ff1bf8a4b1b954e9a247e58a15387fe2: Status 404 returned error can't find the container with id ccdea4af36ba3764fe9ab4be71941d84ff1bf8a4b1b954e9a247e58a15387fe2
Sep 17 22:03:36.164191 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:36.164172    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:03:36.617345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:36.617304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-tflrx" event={"ID":"167219ff-770c-4269-a766-000d8def4b34","Type":"ContainerStarted","Data":"ccdea4af36ba3764fe9ab4be71941d84ff1bf8a4b1b954e9a247e58a15387fe2"}
Sep 17 22:03:53.677539 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:53.677501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-tflrx" event={"ID":"167219ff-770c-4269-a766-000d8def4b34","Type":"ContainerStarted","Data":"3130ab543aa3a6887d7bd8fc60a9e268198f6afd8963aa010e9841cd3f2c359a"}
Sep 17 22:03:53.678024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:53.677605    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:03:53.697030 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:03:53.696938    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-tflrx" podStartSLOduration=2.417348894 podStartE2EDuration="19.69692407s" totalImagesPullingTime="17.279575176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:53.695434906 +0000 UTC m=+338.363497682" watchObservedRunningTime="2026-09-17 22:03:53.69692407 +0000 UTC m=+338.364986817"
Sep 17 22:04:14.683094 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:14.683013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-tflrx"
Sep 17 22:04:40.105915 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.105880    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c94f9f957-spw7w"]
Sep 17 22:04:40.107968 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.107951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.121457 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.121430    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c94f9f957-spw7w"]
Sep 17 22:04:40.252777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-oauth-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.252777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-trusted-ca-bundle\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.252990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.252990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.252990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-service-ca\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.252990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-oauth-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.253118 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.252998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsk2s\" (UniqueName: \"kubernetes.io/projected/038e29b7-e3eb-4676-8fbd-891d72c58ff6-kube-api-access-nsk2s\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-trusted-ca-bundle\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-service-ca\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-oauth-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsk2s\" (UniqueName: \"kubernetes.io/projected/038e29b7-e3eb-4676-8fbd-891d72c58ff6-kube-api-access-nsk2s\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.354643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.354529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-oauth-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.355265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.355216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-service-ca\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.355265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.355255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.355404 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.355255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-oauth-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.355404 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.355326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/038e29b7-e3eb-4676-8fbd-891d72c58ff6-trusted-ca-bundle\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.356938 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.356885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-oauth-config\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.357044 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.357026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/038e29b7-e3eb-4676-8fbd-891d72c58ff6-console-serving-cert\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.368520 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.368490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsk2s\" (UniqueName: \"kubernetes.io/projected/038e29b7-e3eb-4676-8fbd-891d72c58ff6-kube-api-access-nsk2s\") pod \"console-6c94f9f957-spw7w\" (UID: \"038e29b7-e3eb-4676-8fbd-891d72c58ff6\") " pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.416566 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.416517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:40.687129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.687038    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c94f9f957-spw7w"]
Sep 17 22:04:40.689820 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:04:40.689788    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038e29b7_e3eb_4676_8fbd_891d72c58ff6.slice/crio-0962be73c6fd5ae7981aea45508b349783e0bbba565883179339f6d3401d9819 WatchSource:0}: Error finding container 0962be73c6fd5ae7981aea45508b349783e0bbba565883179339f6d3401d9819: Status 404 returned error can't find the container with id 0962be73c6fd5ae7981aea45508b349783e0bbba565883179339f6d3401d9819
Sep 17 22:04:40.837925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.837890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f9f957-spw7w" event={"ID":"038e29b7-e3eb-4676-8fbd-891d72c58ff6","Type":"ContainerStarted","Data":"43c77c76767977f10247d051d200a3020b8200a62fe0bad29d944072a42fd4b2"}
Sep 17 22:04:40.838115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.837941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f9f957-spw7w" event={"ID":"038e29b7-e3eb-4676-8fbd-891d72c58ff6","Type":"ContainerStarted","Data":"0962be73c6fd5ae7981aea45508b349783e0bbba565883179339f6d3401d9819"}
Sep 17 22:04:40.857075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:40.857006    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c94f9f957-spw7w" podStartSLOduration=0.856985555 podStartE2EDuration="856.985555ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:40.855969706 +0000 UTC m=+385.524032453" watchObservedRunningTime="2026-09-17 22:04:40.856985555 +0000 UTC m=+385.525048303"
Sep 17 22:04:50.417518 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:50.417475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:50.418077 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:50.417539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:50.422301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:50.422275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:50.878881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:50.878853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c94f9f957-spw7w"
Sep 17 22:04:50.933143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:50.933115    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:04:51.378368 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.378337    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"]
Sep 17 22:04:51.380779 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.380761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:51.384472 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.384445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 22:04:51.384732 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.384705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-f7l4g\""
Sep 17 22:04:51.384855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.384607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:04:51.385443 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.385425    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:04:51.400580 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.400540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"]
Sep 17 22:04:51.541809 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.541758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llphm\" (UniqueName: \"kubernetes.io/projected/7b51e998-3ced-444a-9e03-a562fc35977f-kube-api-access-llphm\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:51.542282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.541838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:51.642679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.642596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:51.642679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.642665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llphm\" (UniqueName: \"kubernetes.io/projected/7b51e998-3ced-444a-9e03-a562fc35977f-kube-api-access-llphm\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:51.642853 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:04:51.642753    2582 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 22:04:51.642853 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:04:51.642832    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert podName:7b51e998-3ced-444a-9e03-a562fc35977f nodeName:}" failed. No retries permitted until 2026-09-17 22:04:52.142811431 +0000 UTC m=+396.810874161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert") pod "llmisvc-controller-manager-645f9c8fd6-hnklk" (UID: "7b51e998-3ced-444a-9e03-a562fc35977f") : secret "llmisvc-webhook-server-cert" not found
Sep 17 22:04:51.652898 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:51.652868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llphm\" (UniqueName: \"kubernetes.io/projected/7b51e998-3ced-444a-9e03-a562fc35977f-kube-api-access-llphm\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:52.147208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:52.147162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:52.149658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:52.149635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b51e998-3ced-444a-9e03-a562fc35977f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-hnklk\" (UID: \"7b51e998-3ced-444a-9e03-a562fc35977f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:52.292874 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:52.292817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:52.557934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:52.557866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"]
Sep 17 22:04:52.560391 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:04:52.560366    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b51e998_3ced_444a_9e03_a562fc35977f.slice/crio-345540413febea8d42686cdeca45f478e9f6116277ffc6f6c2f44de301780299 WatchSource:0}: Error finding container 345540413febea8d42686cdeca45f478e9f6116277ffc6f6c2f44de301780299: Status 404 returned error can't find the container with id 345540413febea8d42686cdeca45f478e9f6116277ffc6f6c2f44de301780299
Sep 17 22:04:52.883992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:52.883912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk" event={"ID":"7b51e998-3ced-444a-9e03-a562fc35977f","Type":"ContainerStarted","Data":"345540413febea8d42686cdeca45f478e9f6116277ffc6f6c2f44de301780299"}
Sep 17 22:04:55.900164 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:55.900130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk" event={"ID":"7b51e998-3ced-444a-9e03-a562fc35977f","Type":"ContainerStarted","Data":"c5d2252e80f6a80e708a79e179a24b7c30b8abebd35c827ec7d6d6ba8044c219"}
Sep 17 22:04:55.900576 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:55.900361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:04:55.919010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:04:55.918963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk" podStartSLOduration=2.4520944399999998 podStartE2EDuration="4.918948511s" totalImagesPullingTime="2.466854071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:55.91673241 +0000 UTC m=+400.584795182" watchObservedRunningTime="2026-09-17 22:04:55.918948511 +0000 UTC m=+400.587011257"
Sep 17 22:05:15.953853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:15.953790    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86d6d76ccc-wwwdl" podUID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" containerName="console" containerID="cri-o://c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817" gracePeriod=15
Sep 17 22:05:16.188216 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.188192    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:05:16.234920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.234920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.234920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234917    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.235163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.235163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.235163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.234987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.235163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.235019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvnkf\" (UniqueName: \"kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf\") pod \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\" (UID: \"e6a97bec-0b9e-416f-a02c-ebe55b22ef08\") "
Sep 17 22:05:16.235416 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.235385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca" (OuterVolumeSpecName: "service-ca") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:16.235416 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.235407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config" (OuterVolumeSpecName: "console-config") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:16.235567 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.235468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:16.235567 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.235471    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:16.237260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.237229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:16.237352 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.237293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:16.237352 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.237312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf" (OuterVolumeSpecName: "kube-api-access-fvnkf") pod "e6a97bec-0b9e-416f-a02c-ebe55b22ef08" (UID: "e6a97bec-0b9e-416f-a02c-ebe55b22ef08"). InnerVolumeSpecName "kube-api-access-fvnkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:16.336125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336086    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336119    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-trusted-ca-bundle\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336133    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-oauth-serving-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336351 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336145    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnkf\" (UniqueName: \"kubernetes.io/projected/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-kube-api-access-fvnkf\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336351 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336156    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-service-ca\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336351 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336167    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-oauth-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.336351 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.336177    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6a97bec-0b9e-416f-a02c-ebe55b22ef08-console-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:05:16.975660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.975619    2582 generic.go:356] "Generic (PLEG): container finished" podID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" containerID="c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817" exitCode=0
Sep 17 22:05:16.976072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.975671    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d6d76ccc-wwwdl"
Sep 17 22:05:16.976072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.975698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d6d76ccc-wwwdl" event={"ID":"e6a97bec-0b9e-416f-a02c-ebe55b22ef08","Type":"ContainerDied","Data":"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"}
Sep 17 22:05:16.976072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.975745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d6d76ccc-wwwdl" event={"ID":"e6a97bec-0b9e-416f-a02c-ebe55b22ef08","Type":"ContainerDied","Data":"3eec0f40280cacc610b19188ad1f6fa9dfd0e3581c8e94a950aa02ff6e413794"}
Sep 17 22:05:16.976072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.975766    2582 scope.go:117] "RemoveContainer" containerID="c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"
Sep 17 22:05:16.983939 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.983917    2582 scope.go:117] "RemoveContainer" containerID="c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"
Sep 17 22:05:16.984240 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:05:16.984218    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817\": container with ID starting with c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817 not found: ID does not exist" containerID="c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"
Sep 17 22:05:16.984302 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.984253    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817"} err="failed to get container status \"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817\": rpc error: code = NotFound desc = could not find container \"c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817\": container with ID starting with c312764409acf2a86922c2cb5aab295c34ffd5cc9d74d059c8c2e91f0c622817 not found: ID does not exist"
Sep 17 22:05:16.996986 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:16.996960    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:05:17.001048 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:17.001021    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d6d76ccc-wwwdl"]
Sep 17 22:05:17.472412 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:17.472381    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" path="/var/lib/kubelet/pods/e6a97bec-0b9e-416f-a02c-ebe55b22ef08/volumes"
Sep 17 22:05:26.906825 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:05:26.906788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-hnklk"
Sep 17 22:07:44.017826 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.017791    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:44.018288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.018028    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" containerName="console"
Sep 17 22:07:44.018288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.018037    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" containerName="console"
Sep 17 22:07:44.018288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.018087    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a97bec-0b9e-416f-a02c-ebe55b22ef08" containerName="console"
Sep 17 22:07:44.019657 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.019641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.022194 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.022173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 22:07:44.022339 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.022266    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:07:44.028746 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.028724    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:44.108764 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.108725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.108934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.108783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgxv\" (UniqueName: \"kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.209261 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.209228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgxv\" (UniqueName: \"kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.209425 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.209286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.209612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.209593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.218455 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.218431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgxv\" (UniqueName: \"kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv\") pod \"seaweedfs-tls-custom-6644bf5d6c-59pgd\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.328243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.328213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:44.596485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:44.596411    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:44.599653 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:07:44.599623    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261ac2f9_39bf_4b35_bf5c_4e9894858918.slice/crio-28989a8fb6173e27d7cd5ecd7d08148f4c2ac3536d31c7db8c4f32d52b5ca65c WatchSource:0}: Error finding container 28989a8fb6173e27d7cd5ecd7d08148f4c2ac3536d31c7db8c4f32d52b5ca65c: Status 404 returned error can't find the container with id 28989a8fb6173e27d7cd5ecd7d08148f4c2ac3536d31c7db8c4f32d52b5ca65c
Sep 17 22:07:45.486982 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:45.486944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" event={"ID":"261ac2f9-39bf-4b35-bf5c-4e9894858918","Type":"ContainerStarted","Data":"28989a8fb6173e27d7cd5ecd7d08148f4c2ac3536d31c7db8c4f32d52b5ca65c"}
Sep 17 22:07:47.495192 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:47.495098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" event={"ID":"261ac2f9-39bf-4b35-bf5c-4e9894858918","Type":"ContainerStarted","Data":"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"}
Sep 17 22:07:47.511096 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:47.511049    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" podStartSLOduration=0.875537678 podStartE2EDuration="3.511032766s" totalImagesPullingTime="2.635495088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:47.510603274 +0000 UTC m=+572.178666040" watchObservedRunningTime="2026-09-17 22:07:47.511032766 +0000 UTC m=+572.179095513"
Sep 17 22:07:48.882079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:48.882040    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:49.501809 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:49.501764    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" podUID="261ac2f9-39bf-4b35-bf5c-4e9894858918" containerName="seaweedfs-tls-custom" containerID="cri-o://f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83" gracePeriod=30
Sep 17 22:07:50.736453 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.736428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:50.861594 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.861538    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data\") pod \"261ac2f9-39bf-4b35-bf5c-4e9894858918\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") "
Sep 17 22:07:50.861787 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.861621    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvgxv\" (UniqueName: \"kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv\") pod \"261ac2f9-39bf-4b35-bf5c-4e9894858918\" (UID: \"261ac2f9-39bf-4b35-bf5c-4e9894858918\") "
Sep 17 22:07:50.862763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.862737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data" (OuterVolumeSpecName: "data") pod "261ac2f9-39bf-4b35-bf5c-4e9894858918" (UID: "261ac2f9-39bf-4b35-bf5c-4e9894858918"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:50.863826 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.863793    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv" (OuterVolumeSpecName: "kube-api-access-dvgxv") pod "261ac2f9-39bf-4b35-bf5c-4e9894858918" (UID: "261ac2f9-39bf-4b35-bf5c-4e9894858918"). InnerVolumeSpecName "kube-api-access-dvgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:50.962216 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.962158    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvgxv\" (UniqueName: \"kubernetes.io/projected/261ac2f9-39bf-4b35-bf5c-4e9894858918-kube-api-access-dvgxv\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:07:50.962216 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:50.962207    2582 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/261ac2f9-39bf-4b35-bf5c-4e9894858918-data\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:07:51.509978 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.509942    2582 generic.go:356] "Generic (PLEG): container finished" podID="261ac2f9-39bf-4b35-bf5c-4e9894858918" containerID="f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83" exitCode=0
Sep 17 22:07:51.510141 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.509985    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"
Sep 17 22:07:51.510141 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.510030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" event={"ID":"261ac2f9-39bf-4b35-bf5c-4e9894858918","Type":"ContainerDied","Data":"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"}
Sep 17 22:07:51.510141 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.510074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd" event={"ID":"261ac2f9-39bf-4b35-bf5c-4e9894858918","Type":"ContainerDied","Data":"28989a8fb6173e27d7cd5ecd7d08148f4c2ac3536d31c7db8c4f32d52b5ca65c"}
Sep 17 22:07:51.510141 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.510094    2582 scope.go:117] "RemoveContainer" containerID="f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"
Sep 17 22:07:51.519155 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.519138    2582 scope.go:117] "RemoveContainer" containerID="f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"
Sep 17 22:07:51.519407 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:07:51.519389    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83\": container with ID starting with f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83 not found: ID does not exist" containerID="f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"
Sep 17 22:07:51.519485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.519415    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83"} err="failed to get container status \"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83\": rpc error: code = NotFound desc = could not find container \"f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83\": container with ID starting with f2529622043c2395da070a279aa5f2bb26943579ed6a41903580aee15f41ea83 not found: ID does not exist"
Sep 17 22:07:51.525296 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.525272    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:51.528943 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:51.528918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-59pgd"]
Sep 17 22:07:53.472255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:07:53.472212    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261ac2f9-39bf-4b35-bf5c-4e9894858918" path="/var/lib/kubelet/pods/261ac2f9-39bf-4b35-bf5c-4e9894858918/volumes"
Sep 17 22:08:49.329828 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.329793    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:08:49.332837 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.330045    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261ac2f9-39bf-4b35-bf5c-4e9894858918" containerName="seaweedfs-tls-custom"
Sep 17 22:08:49.332837 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.330056    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="261ac2f9-39bf-4b35-bf5c-4e9894858918" containerName="seaweedfs-tls-custom"
Sep 17 22:08:49.332837 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.330099    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="261ac2f9-39bf-4b35-bf5c-4e9894858918" containerName="seaweedfs-tls-custom"
Sep 17 22:08:49.336238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.336207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.341226 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.341195    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:08:49.341365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.341254    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:08:49.341919 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.341890    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 22:08:49.342090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.341929    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:08:49.342090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.341977    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:08:49.342090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.342001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 22:08:49.342792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.342496    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:08:49.347964 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.347936    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:08:49.352970 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.352947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.353075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.352981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.353075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.353043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.353152 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.353083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.353152 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.353126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25p5n\" (UniqueName: \"kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454219 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25p5n\" (UniqueName: \"kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454741 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.454993 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.454972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.455047 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.455007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.456881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.456857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.463098 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.463076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25p5n\" (UniqueName: \"kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-m5g4l\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.646498 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.646399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:08:49.917933 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.917834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:08:49.920573 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:08:49.920534    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3075b05e_684f_4d28_a599_273825bee0ca.slice/crio-67b1b91c70dda16aabe2f3294045e12be0fbde42d103be1de89cd37623ea9120 WatchSource:0}: Error finding container 67b1b91c70dda16aabe2f3294045e12be0fbde42d103be1de89cd37623ea9120: Status 404 returned error can't find the container with id 67b1b91c70dda16aabe2f3294045e12be0fbde42d103be1de89cd37623ea9120
Sep 17 22:08:49.922759 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:49.922742    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:08:50.711014 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:50.710980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerStarted","Data":"67b1b91c70dda16aabe2f3294045e12be0fbde42d103be1de89cd37623ea9120"}
Sep 17 22:08:53.725348 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:53.725312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerStarted","Data":"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"}
Sep 17 22:08:57.739098 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:57.739065    2582 generic.go:356] "Generic (PLEG): container finished" podID="3075b05e-684f-4d28-a599-273825bee0ca" containerID="f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a" exitCode=0
Sep 17 22:08:57.739470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:08:57.739110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerDied","Data":"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"}
Sep 17 22:09:20.840894 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:20.840861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerStarted","Data":"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"}
Sep 17 22:09:23.854825 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:23.854793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerStarted","Data":"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"}
Sep 17 22:09:23.855194 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:23.854950    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:09:23.876469 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:23.876422    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podStartSLOduration=2.702509157 podStartE2EDuration="34.87640643s" totalImagesPullingTime="28.397243911s" totalInitContainerRuntime="3.776653362s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:23.874479219 +0000 UTC m=+668.542541967" watchObservedRunningTime="2026-09-17 22:09:23.87640643 +0000 UTC m=+668.544469237"
Sep 17 22:09:24.858788 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:24.858760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:09:24.860060 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:24.860027    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:09:25.861963 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:25.861925    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:09:30.866848 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:30.866821    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:09:30.867321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:30.867292    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:09:40.867774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:40.867736    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:09:50.867626 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:09:50.867587    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:00.867747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:00.867655    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:10.867321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:10.867281    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:20.868155 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:20.868116    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:30.867792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:30.867749    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:40.868614 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:40.868577    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:10:49.754594 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:49.754538    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:10:49.754986 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:49.754864    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" containerID="cri-o://562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b" gracePeriod=30
Sep 17 22:10:49.754986 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:49.754892    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kube-rbac-proxy" containerID="cri-o://9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49" gracePeriod=30
Sep 17 22:10:50.155190 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:50.155152    2582 generic.go:356] "Generic (PLEG): container finished" podID="3075b05e-684f-4d28-a599-273825bee0ca" containerID="9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49" exitCode=2
Sep 17 22:10:50.155358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:50.155205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerDied","Data":"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"}
Sep 17 22:10:50.862169 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:50.862123    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 22:10:50.867635 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:50.867608    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 22:10:54.948487 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.948444    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:10:54.951170 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.951152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:54.953910 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.953889    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 22:10:54.953910 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.953888    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 22:10:54.961337 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.961316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:10:54.996255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:54.996233    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:10:55.139862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.139763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert\") pod \"3075b05e-684f-4d28-a599-273825bee0ca\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") "
Sep 17 22:10:55.139862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.139830    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls\") pod \"3075b05e-684f-4d28-a599-273825bee0ca\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") "
Sep 17 22:10:55.139862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.139862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25p5n\" (UniqueName: \"kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n\") pod \"3075b05e-684f-4d28-a599-273825bee0ca\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") "
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.139900    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config\") pod \"3075b05e-684f-4d28-a599-273825bee0ca\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") "
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.139923    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location\") pod \"3075b05e-684f-4d28-a599-273825bee0ca\" (UID: \"3075b05e-684f-4d28-a599-273825bee0ca\") "
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.140143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.140428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140224    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3075b05e-684f-4d28-a599-273825bee0ca" (UID: "3075b05e-684f-4d28-a599-273825bee0ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:55.140428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9lx\" (UniqueName: \"kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.140428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140382    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3075b05e-684f-4d28-a599-273825bee0ca" (UID: "3075b05e-684f-4d28-a599-273825bee0ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:55.140428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3075b05e-684f-4d28-a599-273825bee0ca" (UID: "3075b05e-684f-4d28-a599-273825bee0ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:55.140428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.140410    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:10:55.142226 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.142204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3075b05e-684f-4d28-a599-273825bee0ca" (UID: "3075b05e-684f-4d28-a599-273825bee0ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:55.142280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.142221    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n" (OuterVolumeSpecName: "kube-api-access-25p5n") pod "3075b05e-684f-4d28-a599-273825bee0ca" (UID: "3075b05e-684f-4d28-a599-273825bee0ca"). InnerVolumeSpecName "kube-api-access-25p5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:55.175417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.175378    2582 generic.go:356] "Generic (PLEG): container finished" podID="3075b05e-684f-4d28-a599-273825bee0ca" containerID="562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b" exitCode=0
Sep 17 22:10:55.175589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.175443    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"
Sep 17 22:10:55.175589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.175449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerDied","Data":"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"}
Sep 17 22:10:55.175589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.175515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l" event={"ID":"3075b05e-684f-4d28-a599-273825bee0ca","Type":"ContainerDied","Data":"67b1b91c70dda16aabe2f3294045e12be0fbde42d103be1de89cd37623ea9120"}
Sep 17 22:10:55.175589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.175569    2582 scope.go:117] "RemoveContainer" containerID="9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"
Sep 17 22:10:55.184237 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.184218    2582 scope.go:117] "RemoveContainer" containerID="562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"
Sep 17 22:10:55.191654 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.191631    2582 scope.go:117] "RemoveContainer" containerID="f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"
Sep 17 22:10:55.198111 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.198084    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:10:55.198606 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.198592    2582 scope.go:117] "RemoveContainer" containerID="9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"
Sep 17 22:10:55.198872 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:10:55.198855    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49\": container with ID starting with 9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49 not found: ID does not exist" containerID="9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"
Sep 17 22:10:55.198924 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.198879    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49"} err="failed to get container status \"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49\": rpc error: code = NotFound desc = could not find container \"9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49\": container with ID starting with 9b17478b15bd5122af766598196e7c3f224fce2e05721722153e7e235672ee49 not found: ID does not exist"
Sep 17 22:10:55.198924 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.198896    2582 scope.go:117] "RemoveContainer" containerID="562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"
Sep 17 22:10:55.199095 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:10:55.199079    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b\": container with ID starting with 562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b not found: ID does not exist" containerID="562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"
Sep 17 22:10:55.199145 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.199099    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b"} err="failed to get container status \"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b\": rpc error: code = NotFound desc = could not find container \"562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b\": container with ID starting with 562757491e775791030ac7ff59ce2509f4c0cb23771b5659ef544b51e540218b not found: ID does not exist"
Sep 17 22:10:55.199145 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.199112    2582 scope.go:117] "RemoveContainer" containerID="f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"
Sep 17 22:10:55.199326 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:10:55.199310    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a\": container with ID starting with f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a not found: ID does not exist" containerID="f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"
Sep 17 22:10:55.199370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.199327    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a"} err="failed to get container status \"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a\": rpc error: code = NotFound desc = could not find container \"f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a\": container with ID starting with f17e699c2f997c2d0f323e0a347d1977558eff521c009ceb18d59769bf83905a not found: ID does not exist"
Sep 17 22:10:55.201826 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.201804    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-m5g4l"]
Sep 17 22:10:55.240991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.240960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.240999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9lx\" (UniqueName: \"kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241116    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3075b05e-684f-4d28-a599-273825bee0ca-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:10:55.241342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241126    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25p5n\" (UniqueName: \"kubernetes.io/projected/3075b05e-684f-4d28-a599-273825bee0ca-kube-api-access-25p5n\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:10:55.241342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241142    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3075b05e-684f-4d28-a599-273825bee0ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:10:55.241342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241157    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3075b05e-684f-4d28-a599-273825bee0ca-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:10:55.241593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241848 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.241888 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.241827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.243384 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.243363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.249710 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.249688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9lx\" (UniqueName: \"kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx\") pod \"isvc-predictive-xgboost-predictor-695f978978-nzlt6\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.262652 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.262627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:10:55.472675 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.472599    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3075b05e-684f-4d28-a599-273825bee0ca" path="/var/lib/kubelet/pods/3075b05e-684f-4d28-a599-273825bee0ca/volumes"
Sep 17 22:10:55.533924 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:55.533859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:10:55.536477 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:10:55.536433    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276d8130_2b21_4d02_ac2e_f1a6f5e781f3.slice/crio-989c07dee510d7bd29dc13340f3c1dd3119eeba540eb6ed5b7b3cf0ce44036eb WatchSource:0}: Error finding container 989c07dee510d7bd29dc13340f3c1dd3119eeba540eb6ed5b7b3cf0ce44036eb: Status 404 returned error can't find the container with id 989c07dee510d7bd29dc13340f3c1dd3119eeba540eb6ed5b7b3cf0ce44036eb
Sep 17 22:10:56.181712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:56.181666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerStarted","Data":"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"}
Sep 17 22:10:56.182092 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:56.181784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerStarted","Data":"989c07dee510d7bd29dc13340f3c1dd3119eeba540eb6ed5b7b3cf0ce44036eb"}
Sep 17 22:10:59.192737 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:10:59.192706    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276d8130_2b21_4d02_ac2e_f1a6f5e781f3.slice/crio-conmon-184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 22:10:59.193765 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:59.193741    2582 generic.go:356] "Generic (PLEG): container finished" podID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerID="184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82" exitCode=0
Sep 17 22:10:59.193855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:10:59.193783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerDied","Data":"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"}
Sep 17 22:11:00.201513 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:00.201482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerStarted","Data":"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"}
Sep 17 22:11:00.201909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:00.201542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerStarted","Data":"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"}
Sep 17 22:11:00.201909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:00.201777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:11:00.221512 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:00.221454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podStartSLOduration=2.643355493 podStartE2EDuration="6.221436845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.578081352s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:00.220320226 +0000 UTC m=+764.888382990" watchObservedRunningTime="2026-09-17 22:11:00.221436845 +0000 UTC m=+764.889499592"
Sep 17 22:11:01.205663 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:01.205610    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:11:01.206996 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:01.206967    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:02.209611 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:02.209567    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:07.214158 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:07.214127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:11:07.214664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:07.214635    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:17.215245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:17.215205    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:27.214804 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:27.214765    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:37.214709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:37.214618    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:47.215542 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:47.215495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:54.381176 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381134    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381533    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="storage-initializer"
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381562    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="storage-initializer"
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381574    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container"
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381583    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container"
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381600    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kube-rbac-proxy"
Sep 17 22:11:54.381665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381611    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kube-rbac-proxy"
Sep 17 22:11:54.381853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381688    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kube-rbac-proxy"
Sep 17 22:11:54.381853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.381699    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3075b05e-684f-4d28-a599-273825bee0ca" containerName="kserve-container"
Sep 17 22:11:54.383855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.383837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.386466 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.386445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 22:11:54.386586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.386461    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 22:11:54.394180 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.394156    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:11:54.474218 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.474179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzwp\" (UniqueName: \"kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.474436 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.474248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.474436 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.474323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.574926 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.574881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzwp\" (UniqueName: \"kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.575103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.574943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.575103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.574984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.575103 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:11:54.575047    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 22:11:54.575244 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:11:54.575113    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls podName:6da9ed32-cdb4-417f-8b23-b7cb5091ffc6 nodeName:}" failed. No retries permitted until 2026-09-17 22:11:55.075092591 +0000 UTC m=+819.743155327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-477zs" (UID: "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 22:11:54.575648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.575629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:54.583801 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:54.583772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzwp\" (UniqueName: \"kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:55.083417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:55.079051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:55.085779 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:55.085752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-477zs\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:55.294174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:55.294140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:55.566535 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:55.566508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:11:55.569092 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:11:55.569061    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da9ed32_cdb4_417f_8b23_b7cb5091ffc6.slice/crio-872ff33b35ce78dab5dd7661c2d919be315a996199e65ef2dd081b70bd7b8b71 WatchSource:0}: Error finding container 872ff33b35ce78dab5dd7661c2d919be315a996199e65ef2dd081b70bd7b8b71: Status 404 returned error can't find the container with id 872ff33b35ce78dab5dd7661c2d919be315a996199e65ef2dd081b70bd7b8b71
Sep 17 22:11:56.403842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:56.403810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerStarted","Data":"872ff33b35ce78dab5dd7661c2d919be315a996199e65ef2dd081b70bd7b8b71"}
Sep 17 22:11:57.214820 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.214784    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:11:57.409915 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.409876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerStarted","Data":"c70ed83e23f83ff6404f30ba591729cc93af9424bcb3333b53287e02bf63c9db"}
Sep 17 22:11:57.410103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.409939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerStarted","Data":"2c8e7999b9fd59b1ce3b9acd7b451ee4ed075a89c789918437b870c10162a795"}
Sep 17 22:11:57.410103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.410080    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:57.410260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.410242    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:57.411703 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.411670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:11:57.427900 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:11:57.427852    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" podStartSLOduration=2.415629519 podStartE2EDuration="3.427834994s" totalImagesPullingTime="1.012205475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:57.426849277 +0000 UTC m=+822.094912036" watchObservedRunningTime="2026-09-17 22:11:57.427834994 +0000 UTC m=+822.095897741"
Sep 17 22:12:04.420751 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:04.420719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:12:07.214635 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:07.214600    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:12:17.215752 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:17.215712    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:12:25.544900 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:25.544868    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:12:25.545395 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:25.545090    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" containerID="cri-o://3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42" gracePeriod=30
Sep 17 22:12:25.545395 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:25.545155    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kube-rbac-proxy" containerID="cri-o://86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69" gracePeriod=30
Sep 17 22:12:26.513875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:26.513842    2582 generic.go:356] "Generic (PLEG): container finished" podID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerID="86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69" exitCode=2
Sep 17 22:12:26.514054 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:26.513916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerDied","Data":"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"}
Sep 17 22:12:27.210195 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:27.210148    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 22:12:27.215537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:27.215508    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 22:12:30.734309 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.734270    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:12:30.737631 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.737604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.740237 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.740215    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 22:12:30.740465 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.740451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 22:12:30.748871 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.748846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:12:30.871046 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.871014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxql\" (UniqueName: \"kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.871046 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.871052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.871289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.871099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.871289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.871131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.871289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.871158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972240 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxql\" (UniqueName: \"kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.972859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.972831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.973046 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.973025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.973121 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.973063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.975173 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.975136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.981313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.981287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxql\" (UniqueName: \"kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:30.996874 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:30.996814    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:12:31.049628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.049581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:31.073583 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config\") pod \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") "
Sep 17 22:12:31.073740 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9lx\" (UniqueName: \"kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx\") pod \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") "
Sep 17 22:12:31.073797 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073734    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls\") pod \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") "
Sep 17 22:12:31.073853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert\") pod \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") "
Sep 17 22:12:31.073911 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073879    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location\") pod \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\" (UID: \"276d8130-2b21-4d02-ac2e-f1a6f5e781f3\") "
Sep 17 22:12:31.073911 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.073896    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "276d8130-2b21-4d02-ac2e-f1a6f5e781f3" (UID: "276d8130-2b21-4d02-ac2e-f1a6f5e781f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:31.074100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.074081    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:12:31.074188 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.074163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "276d8130-2b21-4d02-ac2e-f1a6f5e781f3" (UID: "276d8130-2b21-4d02-ac2e-f1a6f5e781f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:31.074242 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.074183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "276d8130-2b21-4d02-ac2e-f1a6f5e781f3" (UID: "276d8130-2b21-4d02-ac2e-f1a6f5e781f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:31.075804 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.075772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "276d8130-2b21-4d02-ac2e-f1a6f5e781f3" (UID: "276d8130-2b21-4d02-ac2e-f1a6f5e781f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:31.075897 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.075817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx" (OuterVolumeSpecName: "kube-api-access-5n9lx") pod "276d8130-2b21-4d02-ac2e-f1a6f5e781f3" (UID: "276d8130-2b21-4d02-ac2e-f1a6f5e781f3"). InnerVolumeSpecName "kube-api-access-5n9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:31.174464 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.174428    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:12:31.174464 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.174459    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9lx\" (UniqueName: \"kubernetes.io/projected/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-kube-api-access-5n9lx\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:12:31.174464 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.174471    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:12:31.174738 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.174481    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276d8130-2b21-4d02-ac2e-f1a6f5e781f3-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:12:31.327564 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.327444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:12:31.329714 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:12:31.329686    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24513c24_f56f_4a3e_945f_8ff096c1bb20.slice/crio-85e96c524b93c4f7f19b0583d9a3fc932010269d5b3434407822789bf486c5b6 WatchSource:0}: Error finding container 85e96c524b93c4f7f19b0583d9a3fc932010269d5b3434407822789bf486c5b6: Status 404 returned error can't find the container with id 85e96c524b93c4f7f19b0583d9a3fc932010269d5b3434407822789bf486c5b6
Sep 17 22:12:31.534475 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.534440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerStarted","Data":"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"}
Sep 17 22:12:31.534680 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.534499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerStarted","Data":"85e96c524b93c4f7f19b0583d9a3fc932010269d5b3434407822789bf486c5b6"}
Sep 17 22:12:31.536600 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.536571    2582 generic.go:356] "Generic (PLEG): container finished" podID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerID="3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42" exitCode=0
Sep 17 22:12:31.536730 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.536644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerDied","Data":"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"}
Sep 17 22:12:31.536730 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.536680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"
Sep 17 22:12:31.536730 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.536697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6" event={"ID":"276d8130-2b21-4d02-ac2e-f1a6f5e781f3","Type":"ContainerDied","Data":"989c07dee510d7bd29dc13340f3c1dd3119eeba540eb6ed5b7b3cf0ce44036eb"}
Sep 17 22:12:31.536882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.536734    2582 scope.go:117] "RemoveContainer" containerID="86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"
Sep 17 22:12:31.545361 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.545340    2582 scope.go:117] "RemoveContainer" containerID="3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"
Sep 17 22:12:31.556517 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.556481    2582 scope.go:117] "RemoveContainer" containerID="184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"
Sep 17 22:12:31.566266 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.566245    2582 scope.go:117] "RemoveContainer" containerID="86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"
Sep 17 22:12:31.566645 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:12:31.566621    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69\": container with ID starting with 86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69 not found: ID does not exist" containerID="86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"
Sep 17 22:12:31.566704 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.566660    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69"} err="failed to get container status \"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69\": rpc error: code = NotFound desc = could not find container \"86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69\": container with ID starting with 86e055dc13114df60561d6cf0ab51895392c60320a2347ab49ea0daee662cb69 not found: ID does not exist"
Sep 17 22:12:31.566704 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.566687    2582 scope.go:117] "RemoveContainer" containerID="3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"
Sep 17 22:12:31.566986 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:12:31.566964    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42\": container with ID starting with 3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42 not found: ID does not exist" containerID="3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"
Sep 17 22:12:31.567053 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.566991    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42"} err="failed to get container status \"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42\": rpc error: code = NotFound desc = could not find container \"3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42\": container with ID starting with 3044c0b4dca6d7d8bfd1b628460d465b70ab3b9361da95c5f8e191ce528add42 not found: ID does not exist"
Sep 17 22:12:31.567053 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.567008    2582 scope.go:117] "RemoveContainer" containerID="184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"
Sep 17 22:12:31.567159 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.567076    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:12:31.567282 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:12:31.567266    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82\": container with ID starting with 184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82 not found: ID does not exist" containerID="184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"
Sep 17 22:12:31.567323 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.567286    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82"} err="failed to get container status \"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82\": rpc error: code = NotFound desc = could not find container \"184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82\": container with ID starting with 184dcfc8f9dddafab60364d0e6e592f522eefe1246ecfe4a980d7fd9bf231b82 not found: ID does not exist"
Sep 17 22:12:31.570341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:31.570317    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nzlt6"]
Sep 17 22:12:33.473501 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:33.473465    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" path="/var/lib/kubelet/pods/276d8130-2b21-4d02-ac2e-f1a6f5e781f3/volumes"
Sep 17 22:12:35.552920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:35.552826    2582 generic.go:356] "Generic (PLEG): container finished" podID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerID="5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea" exitCode=0
Sep 17 22:12:35.552920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:35.552885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerDied","Data":"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"}
Sep 17 22:12:36.558102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.558062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerStarted","Data":"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"}
Sep 17 22:12:36.558608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.558135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerStarted","Data":"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"}
Sep 17 22:12:36.558608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.558418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:36.558608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.558474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:36.559794 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.559769    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:12:36.578019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:36.577965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podStartSLOduration=2.790872981 podStartE2EDuration="6.577945348s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.787072367s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:36.576995881 +0000 UTC m=+861.245058652" watchObservedRunningTime="2026-09-17 22:12:36.577945348 +0000 UTC m=+861.246008094"
Sep 17 22:12:37.561685 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:37.561641    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:12:42.565980 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:42.565948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:12:42.566632 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:42.566603    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:12:52.567401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:12:52.567354    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:02.566612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:02.566499    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:12.566987 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:12.566937    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:22.567386 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:22.567343    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:32.567129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:32.567083    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:42.567044 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:42.567005    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:49.455858 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.455727    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-477zs_6da9ed32-cdb4-417f-8b23-b7cb5091ffc6/kserve-container/0.log"
Sep 17 22:13:49.629221 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.629188    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:13:49.629472 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.629445    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kserve-container" containerID="cri-o://2c8e7999b9fd59b1ce3b9acd7b451ee4ed075a89c789918437b870c10162a795" gracePeriod=30
Sep 17 22:13:49.629587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.629490    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kube-rbac-proxy" containerID="cri-o://c70ed83e23f83ff6404f30ba591729cc93af9424bcb3333b53287e02bf63c9db" gracePeriod=30
Sep 17 22:13:49.733941 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.733785    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:13:49.734208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734183    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kube-rbac-proxy"
Sep 17 22:13:49.734208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734209    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kube-rbac-proxy"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734240    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="storage-initializer"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734250    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="storage-initializer"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734267    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734277    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734350    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kserve-container"
Sep 17 22:13:49.734401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.734362    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="276d8130-2b21-4d02-ac2e-f1a6f5e781f3" containerName="kube-rbac-proxy"
Sep 17 22:13:49.737894 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.737867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.740417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.740395    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 22:13:49.740903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.740887    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 22:13:49.746827 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.746729    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:13:49.769163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.769131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.769163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.769167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.769377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.769202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.769377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.769259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.769377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.769282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8499\" (UniqueName: \"kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.826377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.826343    2582 generic.go:356] "Generic (PLEG): container finished" podID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerID="c70ed83e23f83ff6404f30ba591729cc93af9424bcb3333b53287e02bf63c9db" exitCode=2
Sep 17 22:13:49.826377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.826366    2582 generic.go:356] "Generic (PLEG): container finished" podID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerID="2c8e7999b9fd59b1ce3b9acd7b451ee4ed075a89c789918437b870c10162a795" exitCode=2
Sep 17 22:13:49.826641 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.826417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerDied","Data":"c70ed83e23f83ff6404f30ba591729cc93af9424bcb3333b53287e02bf63c9db"}
Sep 17 22:13:49.826641 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.826471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerDied","Data":"2c8e7999b9fd59b1ce3b9acd7b451ee4ed075a89c789918437b870c10162a795"}
Sep 17 22:13:49.869674 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.869643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.869833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.869695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8499\" (UniqueName: \"kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.869833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.869727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.869833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.869745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.869995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.869874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.870365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.870294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.870490 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.870396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.870490 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.870439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.872466 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.872448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.874845 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.874831    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:13:49.878238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.878219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8499\" (UniqueName: \"kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s685j\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:49.970356 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.970317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpzwp\" (UniqueName: \"kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp\") pod \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") "
Sep 17 22:13:49.970527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.970384    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") pod \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") "
Sep 17 22:13:49.970527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.970409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config\") pod \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\" (UID: \"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6\") "
Sep 17 22:13:49.970823 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.970793    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" (UID: "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:49.972504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.972481    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" (UID: "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:49.972602 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:49.972520    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp" (OuterVolumeSpecName: "kube-api-access-cpzwp") pod "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" (UID: "6da9ed32-cdb4-417f-8b23-b7cb5091ffc6"). InnerVolumeSpecName "kube-api-access-cpzwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:50.051146 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.051050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:13:50.071451 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.071420    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpzwp\" (UniqueName: \"kubernetes.io/projected/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-api-access-cpzwp\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:13:50.071451 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.071449    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:13:50.071683 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.071459    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:13:50.323406 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.323376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:13:50.326487 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:13:50.326463    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804946a8_cbd1_41af_92d9_7f4eec864217.slice/crio-6cd08a416d54f4765762778566669d10ad8582d0b2a3f4bf81cdafb43a03c5dc WatchSource:0}: Error finding container 6cd08a416d54f4765762778566669d10ad8582d0b2a3f4bf81cdafb43a03c5dc: Status 404 returned error can't find the container with id 6cd08a416d54f4765762778566669d10ad8582d0b2a3f4bf81cdafb43a03c5dc
Sep 17 22:13:50.831263 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.831229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerStarted","Data":"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"}
Sep 17 22:13:50.831742 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.831297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerStarted","Data":"6cd08a416d54f4765762778566669d10ad8582d0b2a3f4bf81cdafb43a03c5dc"}
Sep 17 22:13:50.832930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.832901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs" event={"ID":"6da9ed32-cdb4-417f-8b23-b7cb5091ffc6","Type":"ContainerDied","Data":"872ff33b35ce78dab5dd7661c2d919be315a996199e65ef2dd081b70bd7b8b71"}
Sep 17 22:13:50.832930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.832930    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"
Sep 17 22:13:50.833075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.832971    2582 scope.go:117] "RemoveContainer" containerID="c70ed83e23f83ff6404f30ba591729cc93af9424bcb3333b53287e02bf63c9db"
Sep 17 22:13:50.842901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.842881    2582 scope.go:117] "RemoveContainer" containerID="2c8e7999b9fd59b1ce3b9acd7b451ee4ed075a89c789918437b870c10162a795"
Sep 17 22:13:50.864646 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.864621    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:13:50.870253 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:50.870230    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-477zs"]
Sep 17 22:13:51.468321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:51.468270    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 22:13:51.472431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:13:51.472400    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" path="/var/lib/kubelet/pods/6da9ed32-cdb4-417f-8b23-b7cb5091ffc6/volumes"
Sep 17 22:14:01.472674 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:01.472647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:14:07.895452 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:07.895417    2582 generic.go:356] "Generic (PLEG): container finished" podID="804946a8-cbd1-41af-92d9-7f4eec864217" containerID="04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65" exitCode=0
Sep 17 22:14:07.895854 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:07.895493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerDied","Data":"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"}
Sep 17 22:14:07.896746 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:07.896729    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:14:11.449913 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:11.449879    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:14:11.450397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:11.450174    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container" containerID="cri-o://509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5" gracePeriod=30
Sep 17 22:14:11.450397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:11.450245    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kube-rbac-proxy" containerID="cri-o://10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b" gracePeriod=30
Sep 17 22:14:11.916341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:11.916298    2582 generic.go:356] "Generic (PLEG): container finished" podID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerID="10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b" exitCode=2
Sep 17 22:14:11.916512 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:11.916369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerDied","Data":"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"}
Sep 17 22:14:12.562809 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:12.562762    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 22:14:17.416786 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.416759    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:14:17.489860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.489788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls\") pod \"24513c24-f56f-4a3e-945f-8ff096c1bb20\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") "
Sep 17 22:14:17.489860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.489849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert\") pod \"24513c24-f56f-4a3e-945f-8ff096c1bb20\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") "
Sep 17 22:14:17.490080 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.489887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location\") pod \"24513c24-f56f-4a3e-945f-8ff096c1bb20\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") "
Sep 17 22:14:17.490080 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.489904    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config\") pod \"24513c24-f56f-4a3e-945f-8ff096c1bb20\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") "
Sep 17 22:14:17.490080 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.489987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsxql\" (UniqueName: \"kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql\") pod \"24513c24-f56f-4a3e-945f-8ff096c1bb20\" (UID: \"24513c24-f56f-4a3e-945f-8ff096c1bb20\") "
Sep 17 22:14:17.490397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.490353    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24513c24-f56f-4a3e-945f-8ff096c1bb20" (UID: "24513c24-f56f-4a3e-945f-8ff096c1bb20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:17.490503 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.490388    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24513c24-f56f-4a3e-945f-8ff096c1bb20" (UID: "24513c24-f56f-4a3e-945f-8ff096c1bb20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:17.490503 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.490400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24513c24-f56f-4a3e-945f-8ff096c1bb20" (UID: "24513c24-f56f-4a3e-945f-8ff096c1bb20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:17.492219 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.492187    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24513c24-f56f-4a3e-945f-8ff096c1bb20" (UID: "24513c24-f56f-4a3e-945f-8ff096c1bb20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:17.492339 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.492260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql" (OuterVolumeSpecName: "kube-api-access-qsxql") pod "24513c24-f56f-4a3e-945f-8ff096c1bb20" (UID: "24513c24-f56f-4a3e-945f-8ff096c1bb20"). InnerVolumeSpecName "kube-api-access-qsxql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:17.590714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.590675    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24513c24-f56f-4a3e-945f-8ff096c1bb20-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:14:17.590714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.590713    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:14:17.590937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.590727    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxql\" (UniqueName: \"kubernetes.io/projected/24513c24-f56f-4a3e-945f-8ff096c1bb20-kube-api-access-qsxql\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:14:17.590937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.590741    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24513c24-f56f-4a3e-945f-8ff096c1bb20-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:14:17.590937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.590756    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24513c24-f56f-4a3e-945f-8ff096c1bb20-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:14:17.946811 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.946768    2582 generic.go:356] "Generic (PLEG): container finished" podID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerID="509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5" exitCode=0
Sep 17 22:14:17.947002 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.946844    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"
Sep 17 22:14:17.947002 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.946830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerDied","Data":"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"}
Sep 17 22:14:17.947002 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.946962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv" event={"ID":"24513c24-f56f-4a3e-945f-8ff096c1bb20","Type":"ContainerDied","Data":"85e96c524b93c4f7f19b0583d9a3fc932010269d5b3434407822789bf486c5b6"}
Sep 17 22:14:17.947127 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.947012    2582 scope.go:117] "RemoveContainer" containerID="10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"
Sep 17 22:14:17.957029 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.956943    2582 scope.go:117] "RemoveContainer" containerID="509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"
Sep 17 22:14:17.964699 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.964652    2582 scope.go:117] "RemoveContainer" containerID="5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"
Sep 17 22:14:17.971534 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.971509    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:14:17.972540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.972518    2582 scope.go:117] "RemoveContainer" containerID="10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"
Sep 17 22:14:17.972866 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:14:17.972845    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b\": container with ID starting with 10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b not found: ID does not exist" containerID="10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"
Sep 17 22:14:17.972945 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.972882    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b"} err="failed to get container status \"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b\": rpc error: code = NotFound desc = could not find container \"10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b\": container with ID starting with 10234311e32cbc6c3ce9265c3d5b66b3e8ed9e6baac1db1793a56553de450e1b not found: ID does not exist"
Sep 17 22:14:17.972945 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.972907    2582 scope.go:117] "RemoveContainer" containerID="509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"
Sep 17 22:14:17.973255 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:14:17.973231    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5\": container with ID starting with 509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5 not found: ID does not exist" containerID="509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"
Sep 17 22:14:17.973335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.973265    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5"} err="failed to get container status \"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5\": rpc error: code = NotFound desc = could not find container \"509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5\": container with ID starting with 509c88cc7dc941da2a923f711ceca5ad72a6b2f3bf3728e2642813a8807cbfa5 not found: ID does not exist"
Sep 17 22:14:17.973335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.973325    2582 scope.go:117] "RemoveContainer" containerID="5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"
Sep 17 22:14:17.973755 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:14:17.973731    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea\": container with ID starting with 5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea not found: ID does not exist" containerID="5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"
Sep 17 22:14:17.973840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.973762    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea"} err="failed to get container status \"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea\": rpc error: code = NotFound desc = could not find container \"5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea\": container with ID starting with 5d1607924dfc473791dc83ad2ed3db3e95f180918a29ae361ce026199267c2ea not found: ID does not exist"
Sep 17 22:14:17.975353 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:17.975335    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hw6gv"]
Sep 17 22:14:19.472541 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:14:19.472508    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" path="/var/lib/kubelet/pods/24513c24-f56f-4a3e-945f-8ff096c1bb20/volumes"
Sep 17 22:16:04.385520 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:04.385479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerStarted","Data":"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"}
Sep 17 22:16:04.386088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:04.385539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerStarted","Data":"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"}
Sep 17 22:16:04.386088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:04.385735    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:04.404103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:04.403875    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podStartSLOduration=2.677150256 podStartE2EDuration="2m15.403854718s" totalImagesPullingTime="1m56.087322996s" totalInitContainerRuntime="16.639381466s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:04.403067063 +0000 UTC m=+1069.071129832" watchObservedRunningTime="2026-09-17 22:16:04.403854718 +0000 UTC m=+1069.071917464"
Sep 17 22:16:05.389191 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:05.389156    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:05.390313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:05.390277    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 22:16:06.392731 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:06.392687    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 22:16:11.398301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:11.397537    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:11.402912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:11.402854    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 22:16:21.398771 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:21.398740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:36.180855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.180815    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:16:36.181354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.181084    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container" containerID="cri-o://da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e" gracePeriod=30
Sep 17 22:16:36.181354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.181156    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kube-rbac-proxy" containerID="cri-o://34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0" gracePeriod=30
Sep 17 22:16:36.392849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.392797    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 22:16:36.503492 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.503411    2582 generic.go:356] "Generic (PLEG): container finished" podID="804946a8-cbd1-41af-92d9-7f4eec864217" containerID="34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0" exitCode=2
Sep 17 22:16:36.503492 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:36.503468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerDied","Data":"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"}
Sep 17 22:16:37.774094 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.774063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:37.895019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.894917    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config\") pod \"804946a8-cbd1-41af-92d9-7f4eec864217\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") "
Sep 17 22:16:37.895019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.894962    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location\") pod \"804946a8-cbd1-41af-92d9-7f4eec864217\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") "
Sep 17 22:16:37.895019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.894995    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8499\" (UniqueName: \"kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499\") pod \"804946a8-cbd1-41af-92d9-7f4eec864217\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") "
Sep 17 22:16:37.895283 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.895027    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert\") pod \"804946a8-cbd1-41af-92d9-7f4eec864217\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") "
Sep 17 22:16:37.895283 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.895070    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls\") pod \"804946a8-cbd1-41af-92d9-7f4eec864217\" (UID: \"804946a8-cbd1-41af-92d9-7f4eec864217\") "
Sep 17 22:16:37.895425 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.895394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "804946a8-cbd1-41af-92d9-7f4eec864217" (UID: "804946a8-cbd1-41af-92d9-7f4eec864217"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:37.895493 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.895410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "804946a8-cbd1-41af-92d9-7f4eec864217" (UID: "804946a8-cbd1-41af-92d9-7f4eec864217"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:37.897214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.897187    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "804946a8-cbd1-41af-92d9-7f4eec864217" (UID: "804946a8-cbd1-41af-92d9-7f4eec864217"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:16:37.897322 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.897277    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499" (OuterVolumeSpecName: "kube-api-access-f8499") pod "804946a8-cbd1-41af-92d9-7f4eec864217" (UID: "804946a8-cbd1-41af-92d9-7f4eec864217"). InnerVolumeSpecName "kube-api-access-f8499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:16:37.899575 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.899530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "804946a8-cbd1-41af-92d9-7f4eec864217" (UID: "804946a8-cbd1-41af-92d9-7f4eec864217"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:16:37.996303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.996265    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/804946a8-cbd1-41af-92d9-7f4eec864217-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:16:37.996303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.996295    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:16:37.996303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.996308    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/804946a8-cbd1-41af-92d9-7f4eec864217-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:16:37.996534 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.996321    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8499\" (UniqueName: \"kubernetes.io/projected/804946a8-cbd1-41af-92d9-7f4eec864217-kube-api-access-f8499\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:16:37.996534 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:37.996333    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/804946a8-cbd1-41af-92d9-7f4eec864217-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:16:38.512668 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.512631    2582 generic.go:356] "Generic (PLEG): container finished" podID="804946a8-cbd1-41af-92d9-7f4eec864217" containerID="da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e" exitCode=0
Sep 17 22:16:38.512863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.512688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerDied","Data":"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"}
Sep 17 22:16:38.512863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.512709    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"
Sep 17 22:16:38.512863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.512735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j" event={"ID":"804946a8-cbd1-41af-92d9-7f4eec864217","Type":"ContainerDied","Data":"6cd08a416d54f4765762778566669d10ad8582d0b2a3f4bf81cdafb43a03c5dc"}
Sep 17 22:16:38.512863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.512761    2582 scope.go:117] "RemoveContainer" containerID="34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"
Sep 17 22:16:38.520758 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.520508    2582 scope.go:117] "RemoveContainer" containerID="da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"
Sep 17 22:16:38.527843 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.527826    2582 scope.go:117] "RemoveContainer" containerID="04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"
Sep 17 22:16:38.534903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.534883    2582 scope.go:117] "RemoveContainer" containerID="34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"
Sep 17 22:16:38.535166 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:16:38.535151    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0\": container with ID starting with 34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0 not found: ID does not exist" containerID="34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"
Sep 17 22:16:38.535211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.535173    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0"} err="failed to get container status \"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0\": rpc error: code = NotFound desc = could not find container \"34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0\": container with ID starting with 34461a328ba23b106478bc0f465b5ce9e4d06311afcb484dbd9552c12774fbd0 not found: ID does not exist"
Sep 17 22:16:38.535211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.535189    2582 scope.go:117] "RemoveContainer" containerID="da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"
Sep 17 22:16:38.535434 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:16:38.535417    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e\": container with ID starting with da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e not found: ID does not exist" containerID="da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"
Sep 17 22:16:38.535471 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.535440    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e"} err="failed to get container status \"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e\": rpc error: code = NotFound desc = could not find container \"da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e\": container with ID starting with da35343de41d40045aa5592449f7ed2ad20961ee08e6c18f52fc6f53e7cff86e not found: ID does not exist"
Sep 17 22:16:38.535471 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.535453    2582 scope.go:117] "RemoveContainer" containerID="04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"
Sep 17 22:16:38.535679 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:16:38.535664    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65\": container with ID starting with 04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65 not found: ID does not exist" containerID="04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"
Sep 17 22:16:38.535720 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.535683    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65"} err="failed to get container status \"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65\": rpc error: code = NotFound desc = could not find container \"04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65\": container with ID starting with 04e120749a04ef1f6cc6a59f2dd490e7da6f7b9d52f2ed8abc890129e3e13d65 not found: ID does not exist"
Sep 17 22:16:38.538310 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.538288    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:16:38.540889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:38.540868    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s685j"]
Sep 17 22:16:39.472049 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:39.472015    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" path="/var/lib/kubelet/pods/804946a8-cbd1-41af-92d9-7f4eec864217/volumes"
Sep 17 22:16:41.380747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380715    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380944    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380955    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380966    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="storage-initializer"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380972    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="storage-initializer"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380983    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380988    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.380998    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381004    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381013    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381019    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381024    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381029    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="storage-initializer"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381042    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="storage-initializer"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381050    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381055    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381093    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381099    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da9ed32-cdb4-417f-8b23-b7cb5091ffc6" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381105    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kserve-container"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381111    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kube-rbac-proxy"
Sep 17 22:16:41.381107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="804946a8-cbd1-41af-92d9-7f4eec864217" containerName="kserve-container"
Sep 17 22:16:41.381766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.381122    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="24513c24-f56f-4a3e-945f-8ff096c1bb20" containerName="kube-rbac-proxy"
Sep 17 22:16:41.403067 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.403035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:16:41.403220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.403170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.406177 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406153    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 22:16:41.406334 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406168    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:16:41.406334 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406168    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:16:41.406463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406162    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:16:41.406463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406315    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:16:41.406584 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.406563    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:16:41.407367 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.407349    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:16:41.522228 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.522200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.522407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.522251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.522407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.522296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.522407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.522370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.522407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.522398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.623743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.623700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.623743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.623746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624001 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.623932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624001 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.623973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624109 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.624015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624217 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.624183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624525 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.624503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.624632 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.624569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.626412 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.626393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.632875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.632818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-q89tw\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:41.713665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:41.713624    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:16:42.010470 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:16:42.010418    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632278cb_a717_44ad_9d37_40b0d1678fd2.slice/crio-bd47a48831d9eef14aae4a6cbe4e46bf60dfdd3c0ba52f65e58273db87d69ff1 WatchSource:0}: Error finding container bd47a48831d9eef14aae4a6cbe4e46bf60dfdd3c0ba52f65e58273db87d69ff1: Status 404 returned error can't find the container with id bd47a48831d9eef14aae4a6cbe4e46bf60dfdd3c0ba52f65e58273db87d69ff1
Sep 17 22:16:42.011645 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:42.011620    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:16:42.535795 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:42.535748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerStarted","Data":"91df7f065f8dc2aa93b31c6cc8371f3d6b81878f73ced58bf91c485cac1612a9"}
Sep 17 22:16:42.536185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:16:42.535868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerStarted","Data":"bd47a48831d9eef14aae4a6cbe4e46bf60dfdd3c0ba52f65e58273db87d69ff1"}
Sep 17 22:17:11.638082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:11.638044    2582 generic.go:356] "Generic (PLEG): container finished" podID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerID="91df7f065f8dc2aa93b31c6cc8371f3d6b81878f73ced58bf91c485cac1612a9" exitCode=0
Sep 17 22:17:11.638608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:11.638125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerDied","Data":"91df7f065f8dc2aa93b31c6cc8371f3d6b81878f73ced58bf91c485cac1612a9"}
Sep 17 22:17:12.644117 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:12.644082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerStarted","Data":"cf2380928756e678879c130c010775c635ceedb61da0535e2b74765ecf66ae98"}
Sep 17 22:17:12.644561 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:12.644166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerStarted","Data":"8eca8999b0bd6b84812951853815f7a51ed13516bf17c24da1b11b5b1d35bd05"}
Sep 17 22:17:12.644561 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:12.644355    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:17:12.665135 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:12.665080    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" podStartSLOduration=2.863012923 podStartE2EDuration="31.665059267s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.802046344s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:12.662019604 +0000 UTC m=+1137.330082397" watchObservedRunningTime="2026-09-17 22:17:12.665059267 +0000 UTC m=+1137.333122016"
Sep 17 22:17:13.647861 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:13.647833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:17:19.657405 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:17:19.657329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:18:46.501677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:46.501643    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:18:57.567076 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:57.567039    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:18:57.568423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:57.567265    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kserve-container" containerID="cri-o://8eca8999b0bd6b84812951853815f7a51ed13516bf17c24da1b11b5b1d35bd05" gracePeriod=30
Sep 17 22:18:57.568423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:57.567293    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kube-rbac-proxy" containerID="cri-o://cf2380928756e678879c130c010775c635ceedb61da0535e2b74765ecf66ae98" gracePeriod=30
Sep 17 22:18:58.024314 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:58.024224    2582 generic.go:356] "Generic (PLEG): container finished" podID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerID="cf2380928756e678879c130c010775c635ceedb61da0535e2b74765ecf66ae98" exitCode=2
Sep 17 22:18:58.024314 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:58.024270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerDied","Data":"cf2380928756e678879c130c010775c635ceedb61da0535e2b74765ecf66ae98"}
Sep 17 22:18:59.037612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.037581    2582 generic.go:356] "Generic (PLEG): container finished" podID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerID="8eca8999b0bd6b84812951853815f7a51ed13516bf17c24da1b11b5b1d35bd05" exitCode=0
Sep 17 22:18:59.037612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.037620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerDied","Data":"8eca8999b0bd6b84812951853815f7a51ed13516bf17c24da1b11b5b1d35bd05"}
Sep 17 22:18:59.102876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.102845    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:18:59.165612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.165501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert\") pod \"632278cb-a717-44ad-9d37-40b0d1678fd2\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") "
Sep 17 22:18:59.165612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.165580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls\") pod \"632278cb-a717-44ad-9d37-40b0d1678fd2\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") "
Sep 17 22:18:59.165612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.165607    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl\") pod \"632278cb-a717-44ad-9d37-40b0d1678fd2\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") "
Sep 17 22:18:59.165897 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.165699    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location\") pod \"632278cb-a717-44ad-9d37-40b0d1678fd2\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") "
Sep 17 22:18:59.165897 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.165723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config\") pod \"632278cb-a717-44ad-9d37-40b0d1678fd2\" (UID: \"632278cb-a717-44ad-9d37-40b0d1678fd2\") "
Sep 17 22:18:59.166047 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.166017    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "632278cb-a717-44ad-9d37-40b0d1678fd2" (UID: "632278cb-a717-44ad-9d37-40b0d1678fd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:18:59.166137 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.166111    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "632278cb-a717-44ad-9d37-40b0d1678fd2" (UID: "632278cb-a717-44ad-9d37-40b0d1678fd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:18:59.167770 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.167748    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl" (OuterVolumeSpecName: "kube-api-access-p5wsl") pod "632278cb-a717-44ad-9d37-40b0d1678fd2" (UID: "632278cb-a717-44ad-9d37-40b0d1678fd2"). InnerVolumeSpecName "kube-api-access-p5wsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:18:59.167858 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.167786    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "632278cb-a717-44ad-9d37-40b0d1678fd2" (UID: "632278cb-a717-44ad-9d37-40b0d1678fd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:18:59.171720 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.171692    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "632278cb-a717-44ad-9d37-40b0d1678fd2" (UID: "632278cb-a717-44ad-9d37-40b0d1678fd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:18:59.266925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.266843    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/632278cb-a717-44ad-9d37-40b0d1678fd2-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:18:59.266925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.266874    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:18:59.266925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.266884    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/632278cb-a717-44ad-9d37-40b0d1678fd2-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:18:59.266925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.266894    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/632278cb-a717-44ad-9d37-40b0d1678fd2-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:18:59.266925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:18:59.266902    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/632278cb-a717-44ad-9d37-40b0d1678fd2-kube-api-access-p5wsl\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:19:00.042802 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.042771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw" event={"ID":"632278cb-a717-44ad-9d37-40b0d1678fd2","Type":"ContainerDied","Data":"bd47a48831d9eef14aae4a6cbe4e46bf60dfdd3c0ba52f65e58273db87d69ff1"}
Sep 17 22:19:00.042802 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.042787    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"
Sep 17 22:19:00.043279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.042849    2582 scope.go:117] "RemoveContainer" containerID="cf2380928756e678879c130c010775c635ceedb61da0535e2b74765ecf66ae98"
Sep 17 22:19:00.050619 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.050601    2582 scope.go:117] "RemoveContainer" containerID="8eca8999b0bd6b84812951853815f7a51ed13516bf17c24da1b11b5b1d35bd05"
Sep 17 22:19:00.058273 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.058252    2582 scope.go:117] "RemoveContainer" containerID="91df7f065f8dc2aa93b31c6cc8371f3d6b81878f73ced58bf91c485cac1612a9"
Sep 17 22:19:00.064721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.064695    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:19:00.069011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:00.068986    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-q89tw"]
Sep 17 22:19:01.473088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:01.473052    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" path="/var/lib/kubelet/pods/632278cb-a717-44ad-9d37-40b0d1678fd2/volumes"
Sep 17 22:19:02.769071 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769032    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769377    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="storage-initializer"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769394    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="storage-initializer"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769409    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kserve-container"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kserve-container"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kube-rbac-proxy"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769442    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kube-rbac-proxy"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769508    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kube-rbac-proxy"
Sep 17 22:19:02.769570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.769517    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="632278cb-a717-44ad-9d37-40b0d1678fd2" containerName="kserve-container"
Sep 17 22:19:02.772909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.772886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.775820 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.775795    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:19:02.775954 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.775894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:19:02.775954 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.775892    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 22:19:02.776081 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.776027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:19:02.776176 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.776151    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:19:02.777055 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.777040    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:19:02.777154 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.777135    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 22:19:02.782333 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.782306    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:19:02.887857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.887819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.887857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.887862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.888070 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.887886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvgc\" (UniqueName: \"kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.888070 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.887979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.888070 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.888005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.988607 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.988541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.988729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.988633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.988729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.988669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.988729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.988689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.988729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.988714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvgc\" (UniqueName: \"kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.989058 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.989035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.989334 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.989307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.989334 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.989318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.991185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.991160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:02.997118 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:02.997086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvgc\" (UniqueName: \"kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:03.084203 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:03.084164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:03.356169 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:03.356089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:19:03.359662 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:19:03.359612    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab71ad35_e2e9_4c9c_8e9a_b834e1da12f6.slice/crio-91d9e1f4e381039ec531bf93c156f9b1aaa90587791b50571d56df941305be79 WatchSource:0}: Error finding container 91d9e1f4e381039ec531bf93c156f9b1aaa90587791b50571d56df941305be79: Status 404 returned error can't find the container with id 91d9e1f4e381039ec531bf93c156f9b1aaa90587791b50571d56df941305be79
Sep 17 22:19:04.059909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:04.059874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerStarted","Data":"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"}
Sep 17 22:19:04.060316 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:04.059931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerStarted","Data":"91d9e1f4e381039ec531bf93c156f9b1aaa90587791b50571d56df941305be79"}
Sep 17 22:19:08.074886 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:08.074857    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerID="ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53" exitCode=0
Sep 17 22:19:08.075234 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:08.074925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerDied","Data":"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"}
Sep 17 22:19:09.081178 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:09.081138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerStarted","Data":"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"}
Sep 17 22:19:09.081597 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:09.081219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerStarted","Data":"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"}
Sep 17 22:19:09.081597 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:09.081446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:09.081597 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:09.081488    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:19:09.099880 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:09.099826    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" podStartSLOduration=2.4645823 podStartE2EDuration="7.099811282s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.635228982s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:19:09.098996343 +0000 UTC m=+1253.767059109" watchObservedRunningTime="2026-09-17 22:19:09.099811282 +0000 UTC m=+1253.767874028"
Sep 17 22:19:15.089420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:19:15.089389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:20:39.504715 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:39.504630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:20:49.712902 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:49.712862    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:20:49.713421 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:49.713208    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kserve-container" containerID="cri-o://aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114" gracePeriod=30
Sep 17 22:20:49.713421 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:49.713239    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kube-rbac-proxy" containerID="cri-o://3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71" gracePeriod=30
Sep 17 22:20:50.085806 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:50.085770    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 22:20:50.442655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:50.442567    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerID="3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71" exitCode=2
Sep 17 22:20:50.442655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:50.442580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerDied","Data":"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"}
Sep 17 22:20:51.360786 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.360759    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:20:51.448431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.448329    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerID="aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114" exitCode=0
Sep 17 22:20:51.448431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.448390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerDied","Data":"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"}
Sep 17 22:20:51.448431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.448403    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"
Sep 17 22:20:51.448735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.448440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk" event={"ID":"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6","Type":"ContainerDied","Data":"91d9e1f4e381039ec531bf93c156f9b1aaa90587791b50571d56df941305be79"}
Sep 17 22:20:51.448735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.448468    2582 scope.go:117] "RemoveContainer" containerID="3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"
Sep 17 22:20:51.456233 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.456212    2582 scope.go:117] "RemoveContainer" containerID="aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"
Sep 17 22:20:51.463341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.463324    2582 scope.go:117] "RemoveContainer" containerID="ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"
Sep 17 22:20:51.470820 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.470802    2582 scope.go:117] "RemoveContainer" containerID="3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"
Sep 17 22:20:51.471096 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:20:51.471077    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71\": container with ID starting with 3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71 not found: ID does not exist" containerID="3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"
Sep 17 22:20:51.471176 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.471112    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71"} err="failed to get container status \"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71\": rpc error: code = NotFound desc = could not find container \"3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71\": container with ID starting with 3adc42525f13a3ed01bdb1aa3b666bb1496b02925030eeb612d837abe3d84e71 not found: ID does not exist"
Sep 17 22:20:51.471176 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.471143    2582 scope.go:117] "RemoveContainer" containerID="aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"
Sep 17 22:20:51.471429 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:20:51.471405    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114\": container with ID starting with aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114 not found: ID does not exist" containerID="aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"
Sep 17 22:20:51.471505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.471435    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114"} err="failed to get container status \"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114\": rpc error: code = NotFound desc = could not find container \"aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114\": container with ID starting with aab91311df8ad043d1ed1b853737c7768cd5d445220f6db767f79a0fba4e1114 not found: ID does not exist"
Sep 17 22:20:51.471505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.471458    2582 scope.go:117] "RemoveContainer" containerID="ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"
Sep 17 22:20:51.471726 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:20:51.471708    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53\": container with ID starting with ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53 not found: ID does not exist" containerID="ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"
Sep 17 22:20:51.471783 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.471731    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53"} err="failed to get container status \"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53\": rpc error: code = NotFound desc = could not find container \"ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53\": container with ID starting with ca8f7cc9b848cf0cfff2c9d9febc29d6dee6cff856fd7cddbccf415de22daf53 not found: ID does not exist"
Sep 17 22:20:51.500245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plvgc\" (UniqueName: \"kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc\") pod \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") "
Sep 17 22:20:51.500245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config\") pod \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") "
Sep 17 22:20:51.500459 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500265    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert\") pod \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") "
Sep 17 22:20:51.500459 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls\") pod \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") "
Sep 17 22:20:51.500459 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location\") pod \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\" (UID: \"ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6\") "
Sep 17 22:20:51.500727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" (UID: "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:51.500727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.500712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" (UID: "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:20:51.502573 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.502536    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" (UID: "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:20:51.502684 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.502653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc" (OuterVolumeSpecName: "kube-api-access-plvgc") pod "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" (UID: "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6"). InnerVolumeSpecName "kube-api-access-plvgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:51.504530 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.504501    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" (UID: "ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:20:51.601100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.601059    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvgc\" (UniqueName: \"kubernetes.io/projected/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-api-access-plvgc\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:20:51.601100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.601092    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:20:51.601100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.601107    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:20:51.601339 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.601120    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:20:51.601339 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.601131    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:20:51.770411 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.770378    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:20:51.776046 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:51.776017    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhjnk"]
Sep 17 22:20:53.472468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:53.472433    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" path="/var/lib/kubelet/pods/ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6/volumes"
Sep 17 22:20:54.909333 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909649    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kserve-container"
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909671    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kserve-container"
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909684    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="storage-initializer"
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909692    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="storage-initializer"
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909719    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kube-rbac-proxy"
Sep 17 22:20:54.909763 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909728    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kube-rbac-proxy"
Sep 17 22:20:54.909947 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909802    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kube-rbac-proxy"
Sep 17 22:20:54.909947 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.909813    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab71ad35-e2e9-4c9c-8e9a-b834e1da12f6" containerName="kserve-container"
Sep 17 22:20:54.915279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.915260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:54.918146 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918113    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 22:20:54.918146 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:20:54.918457 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918138    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:20:54.918457 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918365    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:20:54.918457 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918419    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:20:54.918457 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918447    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 22:20:54.918677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.918477    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:20:54.923058 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:54.923037    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:20:55.025300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.025252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr292\" (UniqueName: \"kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.025300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.025304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.025514 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.025355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.025514 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.025386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.025514 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.025426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.126619 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.126575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr292\" (UniqueName: \"kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.126821 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.126634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.126821 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.126670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.126821 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.126719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.126821 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.126742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.127207 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.127186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.127440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.127421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.127485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.127438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.129058 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.129039    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.136992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.136968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr292\" (UniqueName: \"kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.227066 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.226976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:20:55.497908 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:55.497836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:20:55.500728 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:20:55.500696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fba341_b742_4881_b387_fb6609b5a18c.slice/crio-8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01 WatchSource:0}: Error finding container 8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01: Status 404 returned error can't find the container with id 8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01
Sep 17 22:20:56.467864 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:56.467828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerStarted","Data":"dc97c9a38ab2b916007bef328adef25388d68304a14a49b4a2e41347747e5400"}
Sep 17 22:20:56.468326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:20:56.467882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerStarted","Data":"8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01"}
Sep 17 22:21:25.569421 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:25.569386    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7fba341-b742-4881-b387-fb6609b5a18c" containerID="dc97c9a38ab2b916007bef328adef25388d68304a14a49b4a2e41347747e5400" exitCode=0
Sep 17 22:21:25.569951 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:25.569431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerDied","Data":"dc97c9a38ab2b916007bef328adef25388d68304a14a49b4a2e41347747e5400"}
Sep 17 22:21:26.575351 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:26.575313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerStarted","Data":"259d00788be6dd52d51057c9d04b8828b331bfdd1d377f34df278e7b1181da1b"}
Sep 17 22:21:26.575740 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:26.575405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerStarted","Data":"9d9f92388e3c461803d79db30c751f5f76dbcd8b97e77f1c2340585f67e183c2"}
Sep 17 22:21:26.575740 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:26.575609    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:21:26.595845 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:26.595789    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" podStartSLOduration=2.6753626 podStartE2EDuration="32.595770506s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.920407906s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:21:26.594943387 +0000 UTC m=+1391.263006152" watchObservedRunningTime="2026-09-17 22:21:26.595770506 +0000 UTC m=+1391.263833255"
Sep 17 22:21:27.579404 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:27.579366    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:21:33.588360 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:21:33.588331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:22:57.508268 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:22:57.508223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:23:08.512720 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:08.512671    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.27:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 22:23:11.335092 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:11.335056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:23:11.335521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:11.335335    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kserve-container" containerID="cri-o://9d9f92388e3c461803d79db30c751f5f76dbcd8b97e77f1c2340585f67e183c2" gracePeriod=30
Sep 17 22:23:11.335521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:11.335427    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kube-rbac-proxy" containerID="cri-o://259d00788be6dd52d51057c9d04b8828b331bfdd1d377f34df278e7b1181da1b" gracePeriod=30
Sep 17 22:23:11.955184 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:11.955101    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7fba341-b742-4881-b387-fb6609b5a18c" containerID="259d00788be6dd52d51057c9d04b8828b331bfdd1d377f34df278e7b1181da1b" exitCode=2
Sep 17 22:23:11.955184 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:11.955166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerDied","Data":"259d00788be6dd52d51057c9d04b8828b331bfdd1d377f34df278e7b1181da1b"}
Sep 17 22:23:12.961316 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:12.961283    2582 generic.go:356] "Generic (PLEG): container finished" podID="c7fba341-b742-4881-b387-fb6609b5a18c" containerID="9d9f92388e3c461803d79db30c751f5f76dbcd8b97e77f1c2340585f67e183c2" exitCode=0
Sep 17 22:23:12.961644 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:12.961342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerDied","Data":"9d9f92388e3c461803d79db30c751f5f76dbcd8b97e77f1c2340585f67e183c2"}
Sep 17 22:23:12.961644 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:12.961386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw" event={"ID":"c7fba341-b742-4881-b387-fb6609b5a18c","Type":"ContainerDied","Data":"8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01"}
Sep 17 22:23:12.961644 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:12.961410    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e1dcd69737d544888a2c466601c71d1baccfc1e374ed002b8aa710492a39c01"
Sep 17 22:23:12.972756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:12.972731    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:23:13.149510 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr292\" (UniqueName: \"kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292\") pod \"c7fba341-b742-4881-b387-fb6609b5a18c\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") "
Sep 17 22:23:13.149735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config\") pod \"c7fba341-b742-4881-b387-fb6609b5a18c\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") "
Sep 17 22:23:13.149735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location\") pod \"c7fba341-b742-4881-b387-fb6609b5a18c\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") "
Sep 17 22:23:13.149735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert\") pod \"c7fba341-b742-4881-b387-fb6609b5a18c\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") "
Sep 17 22:23:13.149735 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls\") pod \"c7fba341-b742-4881-b387-fb6609b5a18c\" (UID: \"c7fba341-b742-4881-b387-fb6609b5a18c\") "
Sep 17 22:23:13.149979 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7fba341-b742-4881-b387-fb6609b5a18c" (UID: "c7fba341-b742-4881-b387-fb6609b5a18c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:23:13.150025 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.149966    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7fba341-b742-4881-b387-fb6609b5a18c" (UID: "c7fba341-b742-4881-b387-fb6609b5a18c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:23:13.151806 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.151781    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7fba341-b742-4881-b387-fb6609b5a18c" (UID: "c7fba341-b742-4881-b387-fb6609b5a18c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:23:13.151806 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.151793    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292" (OuterVolumeSpecName: "kube-api-access-qr292") pod "c7fba341-b742-4881-b387-fb6609b5a18c" (UID: "c7fba341-b742-4881-b387-fb6609b5a18c"). InnerVolumeSpecName "kube-api-access-qr292". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:23:13.156197 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.156169    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7fba341-b742-4881-b387-fb6609b5a18c" (UID: "c7fba341-b742-4881-b387-fb6609b5a18c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:13.250495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.250446    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fba341-b742-4881-b387-fb6609b5a18c-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:23:13.250495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.250487    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr292\" (UniqueName: \"kubernetes.io/projected/c7fba341-b742-4881-b387-fb6609b5a18c-kube-api-access-qr292\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:23:13.250495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.250502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:23:13.250767 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.250514    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fba341-b742-4881-b387-fb6609b5a18c-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:23:13.250767 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.250526    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fba341-b742-4881-b387-fb6609b5a18c-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:23:13.966561 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.966514    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"
Sep 17 22:23:13.988431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.988395    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:23:13.992526 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:13.992498    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g6khw"]
Sep 17 22:23:15.476402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:15.476358    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" path="/var/lib/kubelet/pods/c7fba341-b742-4881-b387-fb6609b5a18c/volumes"
Sep 17 22:23:16.557586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557534    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557840    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kserve-container"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kserve-container"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557871    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="storage-initializer"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557876    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="storage-initializer"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557882    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kube-rbac-proxy"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557888    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kube-rbac-proxy"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557929    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kube-rbac-proxy"
Sep 17 22:23:16.557971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.557939    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fba341-b742-4881-b387-fb6609b5a18c" containerName="kserve-container"
Sep 17 22:23:16.562266 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.562246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.565240 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.565213    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:23:16.565380 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.565294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 22:23:16.565380 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.565309    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:23:16.565567 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.565537    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:23:16.565649 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.565623    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:23:16.566354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.566342    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 22:23:16.566399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.566378    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:23:16.571093 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.571070    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:23:16.678103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.678062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.678103 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.678099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.678335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.678141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.678335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.678167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.678335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.678185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zzz\" (UniqueName: \"kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779308 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779308 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779308 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779469 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:23:16.779349    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 22:23:16.779469 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zzz\" (UniqueName: \"kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.779469 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:23:16.779466    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls podName:812250a3-bacd-4476-b0bc-4e091b7fc7c9 nodeName:}" failed. No retries permitted until 2026-09-17 22:23:17.27943652 +0000 UTC m=+1501.947499265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 22:23:16.779649 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.779634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.780049 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.780028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.780087 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.780028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:16.790521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:16.790491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zzz\" (UniqueName: \"kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:17.284739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.284685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:17.287208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.287179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:17.476756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.476726    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:23:17.484235 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.484211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:17.758714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.758688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:23:17.761338 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:23:17.761301    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812250a3_bacd_4476_b0bc_4e091b7fc7c9.slice/crio-9afde3679560e66c8cd467e0d079e0c44d0bb0dd53d2ffb219c04d3708dee0b2 WatchSource:0}: Error finding container 9afde3679560e66c8cd467e0d079e0c44d0bb0dd53d2ffb219c04d3708dee0b2: Status 404 returned error can't find the container with id 9afde3679560e66c8cd467e0d079e0c44d0bb0dd53d2ffb219c04d3708dee0b2
Sep 17 22:23:17.765943 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.765909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:23:17.982352 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.982317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerStarted","Data":"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"}
Sep 17 22:23:17.982513 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:17.982378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerStarted","Data":"9afde3679560e66c8cd467e0d079e0c44d0bb0dd53d2ffb219c04d3708dee0b2"}
Sep 17 22:23:28.019761 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:28.019723    2582 generic.go:356] "Generic (PLEG): container finished" podID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerID="ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910" exitCode=0
Sep 17 22:23:28.020160 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:28.019770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerDied","Data":"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"}
Sep 17 22:23:29.025870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:29.025829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerStarted","Data":"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"}
Sep 17 22:23:29.026356 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:29.025913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerStarted","Data":"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"}
Sep 17 22:23:29.026356 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:29.026135    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:29.045944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:29.045873    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podStartSLOduration=3.7285867919999998 podStartE2EDuration="13.045849003s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.317262211s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:29.043866836 +0000 UTC m=+1513.711929603" watchObservedRunningTime="2026-09-17 22:23:29.045849003 +0000 UTC m=+1513.713911751"
Sep 17 22:23:30.029618 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:30.029574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:30.030818 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:30.030789    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 22:23:31.033301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:31.033266    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 22:23:36.039075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:36.039039    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:36.039736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:36.039704    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 22:23:46.040714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:46.040685    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:23:59.548167 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:59.548131    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:23:59.550899 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:59.548440    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container" containerID="cri-o://8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7" gracePeriod=30
Sep 17 22:23:59.550899 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:23:59.548535    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kube-rbac-proxy" containerID="cri-o://4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5" gracePeriod=30
Sep 17 22:24:00.151973 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:00.151942    2582 generic.go:356] "Generic (PLEG): container finished" podID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerID="4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5" exitCode=2
Sep 17 22:24:00.152211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:00.152024    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerDied","Data":"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"}
Sep 17 22:24:01.034283 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.034238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 22:24:01.490599 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.490571    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:24:01.599378 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location\") pod \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") "
Sep 17 22:24:01.599378 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599384    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zzz\" (UniqueName: \"kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz\") pod \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") "
Sep 17 22:24:01.599639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert\") pod \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") "
Sep 17 22:24:01.599639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599480    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config\") pod \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") "
Sep 17 22:24:01.599639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") pod \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\" (UID: \"812250a3-bacd-4476-b0bc-4e091b7fc7c9\") "
Sep 17 22:24:01.599888 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "812250a3-bacd-4476-b0bc-4e091b7fc7c9" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:01.599947 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.599879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "812250a3-bacd-4476-b0bc-4e091b7fc7c9" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:01.600416 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.600395    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "812250a3-bacd-4476-b0bc-4e091b7fc7c9" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:01.601666 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.601645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "812250a3-bacd-4476-b0bc-4e091b7fc7c9" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:01.601797 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.601777    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz" (OuterVolumeSpecName: "kube-api-access-k6zzz") pod "812250a3-bacd-4476-b0bc-4e091b7fc7c9" (UID: "812250a3-bacd-4476-b0bc-4e091b7fc7c9"). InnerVolumeSpecName "kube-api-access-k6zzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:01.700965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.700907    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:01.700965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.700953    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/812250a3-bacd-4476-b0bc-4e091b7fc7c9-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:01.700965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.700968    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:01.700965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.700982    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zzz\" (UniqueName: \"kubernetes.io/projected/812250a3-bacd-4476-b0bc-4e091b7fc7c9-kube-api-access-k6zzz\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:01.701241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:01.700994    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/812250a3-bacd-4476-b0bc-4e091b7fc7c9-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:02.162250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.162214    2582 generic.go:356] "Generic (PLEG): container finished" podID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerID="8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7" exitCode=0
Sep 17 22:24:02.162781 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.162279    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"
Sep 17 22:24:02.162781 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.162299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerDied","Data":"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"}
Sep 17 22:24:02.162781 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.162358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69" event={"ID":"812250a3-bacd-4476-b0bc-4e091b7fc7c9","Type":"ContainerDied","Data":"9afde3679560e66c8cd467e0d079e0c44d0bb0dd53d2ffb219c04d3708dee0b2"}
Sep 17 22:24:02.162781 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.162384    2582 scope.go:117] "RemoveContainer" containerID="4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"
Sep 17 22:24:02.171211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.171184    2582 scope.go:117] "RemoveContainer" containerID="8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"
Sep 17 22:24:02.178963 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.178944    2582 scope.go:117] "RemoveContainer" containerID="ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"
Sep 17 22:24:02.184538 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.184511    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:24:02.187355 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.187333    2582 scope.go:117] "RemoveContainer" containerID="4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"
Sep 17 22:24:02.187811 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:02.187781    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5\": container with ID starting with 4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5 not found: ID does not exist" containerID="4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"
Sep 17 22:24:02.187929 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.187824    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5"} err="failed to get container status \"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5\": rpc error: code = NotFound desc = could not find container \"4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5\": container with ID starting with 4f7a24898a3061ed2b2c2bc206a4c8eb1d82ba2190af502d80b9e97b7a5b46e5 not found: ID does not exist"
Sep 17 22:24:02.187929 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.187853    2582 scope.go:117] "RemoveContainer" containerID="8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"
Sep 17 22:24:02.188173 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:02.188154    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7\": container with ID starting with 8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7 not found: ID does not exist" containerID="8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"
Sep 17 22:24:02.188241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.188183    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7"} err="failed to get container status \"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7\": rpc error: code = NotFound desc = could not find container \"8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7\": container with ID starting with 8ac3b534913f464571c4bb3b10326787464a4cccff29a30cae9537b68ad96fd7 not found: ID does not exist"
Sep 17 22:24:02.188241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.188206    2582 scope.go:117] "RemoveContainer" containerID="ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"
Sep 17 22:24:02.188569 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:02.188534    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910\": container with ID starting with ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910 not found: ID does not exist" containerID="ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"
Sep 17 22:24:02.188639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.188578    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910"} err="failed to get container status \"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910\": rpc error: code = NotFound desc = could not find container \"ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910\": container with ID starting with ccc1b6c540d5156e2086f67af324696138542df907e1bb56ca76ac82dd39d910 not found: ID does not exist"
Sep 17 22:24:02.188756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:02.188739    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-frc69"]
Sep 17 22:24:03.472604 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:03.472543    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" path="/var/lib/kubelet/pods/812250a3-bacd-4476-b0bc-4e091b7fc7c9/volumes"
Sep 17 22:24:04.755762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.755727    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756000    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kube-rbac-proxy"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756011    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kube-rbac-proxy"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756020    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="storage-initializer"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756026    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="storage-initializer"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756033    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756039    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756090    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kube-rbac-proxy"
Sep 17 22:24:04.756131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.756098    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="812250a3-bacd-4476-b0bc-4e091b7fc7c9" containerName="kserve-container"
Sep 17 22:24:04.760589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.760562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.763342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.763321    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 22:24:04.763479 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.763362    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:24:04.763633 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.763618    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:24:04.763697 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.763652    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 22:24:04.764652 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.764631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:24:04.764876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.764863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:24:04.764965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.764863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:24:04.768903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.768878    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:04.826134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.826098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.826134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.826138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.826372 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.826170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.826372 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.826189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.826372 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.826240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlzm\" (UniqueName: \"kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.926739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.926701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.926739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.926751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.926795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.926817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.926859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlzm\" (UniqueName: \"kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927011 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:04.926969    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 22:24:04.927208 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:04.927041    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls podName:8bf8fc9e-1949-441f-82e1-dd11fbd70baa nodeName:}" failed. No retries permitted until 2026-09-17 22:24:05.427016978 +0000 UTC m=+1550.095079710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 22:24:04.927208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.927162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927533 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.927514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.927620 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.927598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:04.935655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:04.935623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlzm\" (UniqueName: \"kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:05.431128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:05.431083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:05.433659 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:05.433636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:05.672188 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:05.672149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:05.942967 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:05.942891    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:05.945869 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:24:05.945842    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf8fc9e_1949_441f_82e1_dd11fbd70baa.slice/crio-6562631bd94341a9122f9969086ca7e2fffbc20a51c430490d354fe78ff2f339 WatchSource:0}: Error finding container 6562631bd94341a9122f9969086ca7e2fffbc20a51c430490d354fe78ff2f339: Status 404 returned error can't find the container with id 6562631bd94341a9122f9969086ca7e2fffbc20a51c430490d354fe78ff2f339
Sep 17 22:24:06.178309 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:06.178266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerStarted","Data":"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"}
Sep 17 22:24:06.178507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:06.178341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerStarted","Data":"6562631bd94341a9122f9969086ca7e2fffbc20a51c430490d354fe78ff2f339"}
Sep 17 22:24:11.197660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:11.197626    2582 generic.go:356] "Generic (PLEG): container finished" podID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerID="d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87" exitCode=0
Sep 17 22:24:11.198045 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:11.197677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerDied","Data":"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"}
Sep 17 22:24:12.203537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:12.203502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerStarted","Data":"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"}
Sep 17 22:24:12.203920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:12.203578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerStarted","Data":"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"}
Sep 17 22:24:12.203920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:12.203781    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:12.224002 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:12.223944    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podStartSLOduration=3.915267992 podStartE2EDuration="8.22392656s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.308658568s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:12.222653938 +0000 UTC m=+1556.890716701" watchObservedRunningTime="2026-09-17 22:24:12.22392656 +0000 UTC m=+1556.891989307"
Sep 17 22:24:13.206966 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:13.206929    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:13.208356 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:13.208326    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 22:24:14.210382 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:14.210342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 22:24:19.216422 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:19.216385    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:19.217085 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:19.217053    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 22:24:29.217539 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:29.217509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:40.437461 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:40.437421    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:40.438011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:40.437762    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container" containerID="cri-o://8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118" gracePeriod=30
Sep 17 22:24:40.438011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:40.437806    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kube-rbac-proxy" containerID="cri-o://18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb" gracePeriod=30
Sep 17 22:24:41.318093 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:41.318058    2582 generic.go:356] "Generic (PLEG): container finished" podID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerID="18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb" exitCode=2
Sep 17 22:24:41.318276 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:41.318101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerDied","Data":"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"}
Sep 17 22:24:41.991886 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:41.991859    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:42.118179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert\") pod \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") "
Sep 17 22:24:42.118179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location\") pod \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") "
Sep 17 22:24:42.118179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config\") pod \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") "
Sep 17 22:24:42.118470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118275    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") pod \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") "
Sep 17 22:24:42.118470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzlzm\" (UniqueName: \"kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm\") pod \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\" (UID: \"8bf8fc9e-1949-441f-82e1-dd11fbd70baa\") "
Sep 17 22:24:42.118602 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118576    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bf8fc9e-1949-441f-82e1-dd11fbd70baa" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:42.118660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.118591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bf8fc9e-1949-441f-82e1-dd11fbd70baa" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:42.119137 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.119116    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bf8fc9e-1949-441f-82e1-dd11fbd70baa" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:42.120436 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.120417    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bf8fc9e-1949-441f-82e1-dd11fbd70baa" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:42.120497 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.120418    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm" (OuterVolumeSpecName: "kube-api-access-tzlzm") pod "8bf8fc9e-1949-441f-82e1-dd11fbd70baa" (UID: "8bf8fc9e-1949-441f-82e1-dd11fbd70baa"). InnerVolumeSpecName "kube-api-access-tzlzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:42.218930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.218889    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:42.218930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.218921    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:42.218930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.218932    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:42.218930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.218940    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzlzm\" (UniqueName: \"kubernetes.io/projected/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-kube-api-access-tzlzm\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:42.219200 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.218949    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8fc9e-1949-441f-82e1-dd11fbd70baa-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:24:42.324508 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.324464    2582 generic.go:356] "Generic (PLEG): container finished" podID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerID="8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118" exitCode=0
Sep 17 22:24:42.324712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.324543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"
Sep 17 22:24:42.324712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.324537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerDied","Data":"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"}
Sep 17 22:24:42.324712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.324692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn" event={"ID":"8bf8fc9e-1949-441f-82e1-dd11fbd70baa","Type":"ContainerDied","Data":"6562631bd94341a9122f9969086ca7e2fffbc20a51c430490d354fe78ff2f339"}
Sep 17 22:24:42.324856 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.324722    2582 scope.go:117] "RemoveContainer" containerID="18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"
Sep 17 22:24:42.332780 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.332755    2582 scope.go:117] "RemoveContainer" containerID="8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"
Sep 17 22:24:42.341494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.341474    2582 scope.go:117] "RemoveContainer" containerID="d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"
Sep 17 22:24:42.351133 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.351094    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:42.352111 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.352095    2582 scope.go:117] "RemoveContainer" containerID="18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"
Sep 17 22:24:42.352448 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:42.352431    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb\": container with ID starting with 18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb not found: ID does not exist" containerID="18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"
Sep 17 22:24:42.352489 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.352457    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb"} err="failed to get container status \"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb\": rpc error: code = NotFound desc = could not find container \"18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb\": container with ID starting with 18fe0404883f2461d0c66f70331146794872ebd9cad28ae6cd0c4b52d35c86bb not found: ID does not exist"
Sep 17 22:24:42.352489 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.352475    2582 scope.go:117] "RemoveContainer" containerID="8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"
Sep 17 22:24:42.352792 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:42.352774    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118\": container with ID starting with 8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118 not found: ID does not exist" containerID="8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"
Sep 17 22:24:42.352869 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.352798    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118"} err="failed to get container status \"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118\": rpc error: code = NotFound desc = could not find container \"8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118\": container with ID starting with 8da8ccd2a52f672e5d85da4c1c0ee338f73ac9480c81a00e46038f207fc44118 not found: ID does not exist"
Sep 17 22:24:42.352869 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.352822    2582 scope.go:117] "RemoveContainer" containerID="d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"
Sep 17 22:24:42.353166 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:24:42.353141    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87\": container with ID starting with d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87 not found: ID does not exist" containerID="d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"
Sep 17 22:24:42.353327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.353172    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87"} err="failed to get container status \"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87\": rpc error: code = NotFound desc = could not find container \"d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87\": container with ID starting with d14cd316b0b21df46dc9c6606cf6ae286374cb95b70edb85565d97e8ed2c7d87 not found: ID does not exist"
Sep 17 22:24:42.354951 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:42.354929    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6xbqn"]
Sep 17 22:24:43.472827 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:43.472789    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" path="/var/lib/kubelet/pods/8bf8fc9e-1949-441f-82e1-dd11fbd70baa/volumes"
Sep 17 22:24:45.727507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727469    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727832    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kube-rbac-proxy"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727850    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kube-rbac-proxy"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727868    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727876    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727899    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="storage-initializer"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727908    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="storage-initializer"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727977    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kserve-container"
Sep 17 22:24:45.728000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.727988    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf8fc9e-1949-441f-82e1-dd11fbd70baa" containerName="kube-rbac-proxy"
Sep 17 22:24:45.732460 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.732439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.735338 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.735314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:24:45.735479 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.735377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 22:24:45.735479 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.735414    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:24:45.735609 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.735527    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 22:24:45.735939 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.735919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:24:45.736777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.736760    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:24:45.736875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.736796    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:24:45.742374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.742350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:24:45.848733 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.848691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmmc\" (UniqueName: \"kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.848907 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.848753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.848907 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.848805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.848907 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.848825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.848907 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.848887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950008 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.949955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmmc\" (UniqueName: \"kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.950886 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.951001 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.950928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.952849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.952828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:45.958730 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:45.958701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmmc\" (UniqueName: \"kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc\") pod \"isvc-canary-create-predictor-5f845bd94d-7tx46\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:46.043426 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:46.043333    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:24:46.317641 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:46.317606    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:24:46.320450 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:24:46.320417    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f4e40a_231d_45f6_bf22_ffa9ea659bbd.slice/crio-d2111cee1227e54b9b440658d1384471b3f7449b6c6091a5f6d3cfd211608199 WatchSource:0}: Error finding container d2111cee1227e54b9b440658d1384471b3f7449b6c6091a5f6d3cfd211608199: Status 404 returned error can't find the container with id d2111cee1227e54b9b440658d1384471b3f7449b6c6091a5f6d3cfd211608199
Sep 17 22:24:46.341586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:46.341560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerStarted","Data":"d2111cee1227e54b9b440658d1384471b3f7449b6c6091a5f6d3cfd211608199"}
Sep 17 22:24:47.349842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:47.349805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerStarted","Data":"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"}
Sep 17 22:24:50.362912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:50.362811    2582 generic.go:356] "Generic (PLEG): container finished" podID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerID="f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d" exitCode=0
Sep 17 22:24:50.362912 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:50.362880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerDied","Data":"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"}
Sep 17 22:24:50.364113 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:24:50.364098    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:25:04.432055 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:04.432020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerStarted","Data":"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"}
Sep 17 22:25:04.432433 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:04.432109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerStarted","Data":"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"}
Sep 17 22:25:04.432433 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:04.432315    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:25:04.451964 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:04.451903    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podStartSLOduration=2.31597822 podStartE2EDuration="19.451887738s" totalImagesPullingTime="13.3985574s" totalInitContainerRuntime="3.737352118s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:04.450011538 +0000 UTC m=+1609.118074306" watchObservedRunningTime="2026-09-17 22:25:04.451887738 +0000 UTC m=+1609.119950484"
Sep 17 22:25:05.436619 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:05.436580    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:25:05.438114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:05.438082    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:06.440981 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:06.440943    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:11.445201 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:11.445166    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:25:11.445742 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:11.445713    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:21.445982 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:21.445940    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:31.446034 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:31.445992    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:41.445825 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:41.445785    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:25:51.446322 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:25:51.446275    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:26:01.446661 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:01.446618    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:26:11.445891 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:11.445845    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:26:17.772061 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:17.772022    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:26:17.772496 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:17.772260    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" containerID="cri-o://154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9" gracePeriod=30
Sep 17 22:26:17.772496 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:17.772346    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kube-rbac-proxy" containerID="cri-o://f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e" gracePeriod=30
Sep 17 22:26:17.772496 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:17.772480    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:26:18.702377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:18.702344    2582 generic.go:356] "Generic (PLEG): container finished" podID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerID="f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e" exitCode=2
Sep 17 22:26:18.702603 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:18.702414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerDied","Data":"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"}
Sep 17 22:26:21.441852 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:21.441792    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:26:23.913656 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:23.913630    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:26:24.012686 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012584    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config\") pod \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") "
Sep 17 22:26:24.012686 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location\") pod \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") "
Sep 17 22:26:24.012686 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012650    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmmmc\" (UniqueName: \"kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc\") pod \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") "
Sep 17 22:26:24.012956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls\") pod \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") "
Sep 17 22:26:24.012956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert\") pod \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\" (UID: \"99f4e40a-231d-45f6-bf22-ffa9ea659bbd\") "
Sep 17 22:26:24.013071 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.012972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99f4e40a-231d-45f6-bf22-ffa9ea659bbd" (UID: "99f4e40a-231d-45f6-bf22-ffa9ea659bbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:24.013128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.013007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99f4e40a-231d-45f6-bf22-ffa9ea659bbd" (UID: "99f4e40a-231d-45f6-bf22-ffa9ea659bbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:24.013178 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.013159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99f4e40a-231d-45f6-bf22-ffa9ea659bbd" (UID: "99f4e40a-231d-45f6-bf22-ffa9ea659bbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:24.014835 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.014812    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99f4e40a-231d-45f6-bf22-ffa9ea659bbd" (UID: "99f4e40a-231d-45f6-bf22-ffa9ea659bbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:26:24.014922 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.014828    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc" (OuterVolumeSpecName: "kube-api-access-dmmmc") pod "99f4e40a-231d-45f6-bf22-ffa9ea659bbd" (UID: "99f4e40a-231d-45f6-bf22-ffa9ea659bbd"). InnerVolumeSpecName "kube-api-access-dmmmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:26:24.113185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.113149    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmmc\" (UniqueName: \"kubernetes.io/projected/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-api-access-dmmmc\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:26:24.113185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.113182    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:26:24.113185 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.113194    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:26:24.113410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.113207    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:26:24.113410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.113219    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f4e40a-231d-45f6-bf22-ffa9ea659bbd-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:26:24.726142 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.726101    2582 generic.go:356] "Generic (PLEG): container finished" podID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerID="154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9" exitCode=139
Sep 17 22:26:24.726313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.726150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerDied","Data":"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"}
Sep 17 22:26:24.726313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.726165    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"
Sep 17 22:26:24.726313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.726213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46" event={"ID":"99f4e40a-231d-45f6-bf22-ffa9ea659bbd","Type":"ContainerDied","Data":"d2111cee1227e54b9b440658d1384471b3f7449b6c6091a5f6d3cfd211608199"}
Sep 17 22:26:24.726313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.726250    2582 scope.go:117] "RemoveContainer" containerID="f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"
Sep 17 22:26:24.734585 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.734539    2582 scope.go:117] "RemoveContainer" containerID="154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"
Sep 17 22:26:24.741963 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.741943    2582 scope.go:117] "RemoveContainer" containerID="f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"
Sep 17 22:26:24.747708 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.747684    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:26:24.750151 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.750118    2582 scope.go:117] "RemoveContainer" containerID="f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"
Sep 17 22:26:24.750444 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:26:24.750425    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e\": container with ID starting with f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e not found: ID does not exist" containerID="f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"
Sep 17 22:26:24.750530 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.750453    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e"} err="failed to get container status \"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e\": rpc error: code = NotFound desc = could not find container \"f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e\": container with ID starting with f4acb80de66c64ffa2efbc9406ead7d44549d0046dd14693ca9fbcc146d2ee5e not found: ID does not exist"
Sep 17 22:26:24.750530 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.750477    2582 scope.go:117] "RemoveContainer" containerID="154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"
Sep 17 22:26:24.750776 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:26:24.750757    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9\": container with ID starting with 154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9 not found: ID does not exist" containerID="154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"
Sep 17 22:26:24.750833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.750781    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9"} err="failed to get container status \"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9\": rpc error: code = NotFound desc = could not find container \"154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9\": container with ID starting with 154a29e4d047b189d3cb0bd9ac5fc93f14cbaa35d54f9dbbf801d77ced69b0e9 not found: ID does not exist"
Sep 17 22:26:24.750833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.750799    2582 scope.go:117] "RemoveContainer" containerID="f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"
Sep 17 22:26:24.751067 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:26:24.751048    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d\": container with ID starting with f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d not found: ID does not exist" containerID="f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"
Sep 17 22:26:24.751127 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.751072    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d"} err="failed to get container status \"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d\": rpc error: code = NotFound desc = could not find container \"f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d\": container with ID starting with f8b7e5cf4d1facd099b12936eeaef46cd3fe8356f26af5748c1cbb2b78d3e73d not found: ID does not exist"
Sep 17 22:26:24.751407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:24.751384    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-7tx46"]
Sep 17 22:26:25.472358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:25.472319    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" path="/var/lib/kubelet/pods/99f4e40a-231d-45f6-bf22-ffa9ea659bbd/volumes"
Sep 17 22:26:29.847685 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847602    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847846    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847858    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847876    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="storage-initializer"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847883    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="storage-initializer"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847888    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kube-rbac-proxy"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847893    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kube-rbac-proxy"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847947    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kserve-container"
Sep 17 22:26:29.848129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.847957    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f4e40a-231d-45f6-bf22-ffa9ea659bbd" containerName="kube-rbac-proxy"
Sep 17 22:26:29.850483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.850466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.853319 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:26:29.853319 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 22:26:29.853507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:26:29.853507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853383    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:26:29.853507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853287    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:26:29.853507 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.853293    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:26:29.854332 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.854314    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:26:29.857218 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.857195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.857309 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.857254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.857355 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.857339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.857411 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.857389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.857473 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.857424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56jd\" (UniqueName: \"kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.860281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.860257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:26:29.958271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.958465 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.958465 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.958605 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56jd\" (UniqueName: \"kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.958605 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.958605 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:26:29.958540    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 22:26:29.958777 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:26:29.958630    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls podName:b54fa823-9293-4b74-a623-89904d8909a8 nodeName:}" failed. No retries permitted until 2026-09-17 22:26:30.458607132 +0000 UTC m=+1695.126669864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls") pod "isvc-sklearn-v2-predictor-555d868545-2lh99" (UID: "b54fa823-9293-4b74-a623-89904d8909a8") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 22:26:29.958890 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.958867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.959217 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.959008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.959217 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.959140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:29.967313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:29.967286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56jd\" (UniqueName: \"kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:30.461007 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.460957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:30.463392 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.463367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-555d868545-2lh99\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:30.761737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.761634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:30.883602 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.882906    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:26:30.886664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.886636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:30.890529 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.890499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:26:30.890807 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.890641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 22:26:30.894487 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.894385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:26:30.918492 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.918458    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:26:30.920752 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.920735    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.923166 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.923142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 22:26:30.933527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.933499    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:26:30.965179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.965179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9bp\" (UniqueName: \"kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.965424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.965424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5268\" (UniqueName: \"kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:30.965424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:30.965424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:30.965424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.965721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:30.965721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:30.965721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:30.965524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.035916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.035711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:26:31.038462 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:26:31.038437    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54fa823_9293_4b74_a623_89904d8909a8.slice/crio-4c9d47d1e70b4fcd936988ed10cce99387b8c13efc9e5552fcd49521482b161b WatchSource:0}: Error finding container 4c9d47d1e70b4fcd936988ed10cce99387b8c13efc9e5552fcd49521482b161b: Status 404 returned error can't find the container with id 4c9d47d1e70b4fcd936988ed10cce99387b8c13efc9e5552fcd49521482b161b
Sep 17 22:26:31.066217 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9bp\" (UniqueName: \"kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.066217 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.066424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5268\" (UniqueName: \"kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066526 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066526 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.066526 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.066724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.066882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.066985 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.067050 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.066985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.067267 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.067248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.067358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.067321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.067358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.067336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.069447 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.069423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.069539 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.069430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.075632 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.075610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9bp\" (UniqueName: \"kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp\") pod \"isvc-canary-promote-predictor-567686cc76-qtxgw\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.075826 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.075804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5268\" (UniqueName: \"kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.200846 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.200803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:31.231260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.231215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:31.475835 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.475802    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:26:31.478410 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:26:31.478385    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a60fae_58ba_4249_b4f8_8d16fbb01725.slice/crio-5397db14d2036fc7530125629cd1843b89fb2c829830386e86fcfbd9fe25d614 WatchSource:0}: Error finding container 5397db14d2036fc7530125629cd1843b89fb2c829830386e86fcfbd9fe25d614: Status 404 returned error can't find the container with id 5397db14d2036fc7530125629cd1843b89fb2c829830386e86fcfbd9fe25d614
Sep 17 22:26:31.511798 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.511738    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:26:31.517488 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:26:31.517457    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b51a82_68a2_4ad9_aefa_dbc71f6f2160.slice/crio-ece1e38a2cef0d963964a45c3413b157124237d5498d83762d7463b1d85b68f0 WatchSource:0}: Error finding container ece1e38a2cef0d963964a45c3413b157124237d5498d83762d7463b1d85b68f0: Status 404 returned error can't find the container with id ece1e38a2cef0d963964a45c3413b157124237d5498d83762d7463b1d85b68f0
Sep 17 22:26:31.757594 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.757525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerStarted","Data":"2e34f50824b92dec1ad5d6a2590aa29a970271f5884d1d9f138dfef52d9654ac"}
Sep 17 22:26:31.757802 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.757637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerStarted","Data":"4c9d47d1e70b4fcd936988ed10cce99387b8c13efc9e5552fcd49521482b161b"}
Sep 17 22:26:31.759463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.759422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerStarted","Data":"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"}
Sep 17 22:26:31.759621 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.759497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerStarted","Data":"5397db14d2036fc7530125629cd1843b89fb2c829830386e86fcfbd9fe25d614"}
Sep 17 22:26:31.761511 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.761474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerStarted","Data":"e636640a9989c4d910abcb4659662a70b2754927de434361eab0b97d5c0aced4"}
Sep 17 22:26:31.761652 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:31.761566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerStarted","Data":"ece1e38a2cef0d963964a45c3413b157124237d5498d83762d7463b1d85b68f0"}
Sep 17 22:26:35.779459 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.779420    2582 generic.go:356] "Generic (PLEG): container finished" podID="b54fa823-9293-4b74-a623-89904d8909a8" containerID="2e34f50824b92dec1ad5d6a2590aa29a970271f5884d1d9f138dfef52d9654ac" exitCode=0
Sep 17 22:26:35.779968 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.779499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerDied","Data":"2e34f50824b92dec1ad5d6a2590aa29a970271f5884d1d9f138dfef52d9654ac"}
Sep 17 22:26:35.781054 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.781027    2582 generic.go:356] "Generic (PLEG): container finished" podID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerID="b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8" exitCode=0
Sep 17 22:26:35.781154 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.781104    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerDied","Data":"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"}
Sep 17 22:26:35.782867 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.782836    2582 generic.go:356] "Generic (PLEG): container finished" podID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerID="e636640a9989c4d910abcb4659662a70b2754927de434361eab0b97d5c0aced4" exitCode=0
Sep 17 22:26:35.782943 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:35.782872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerDied","Data":"e636640a9989c4d910abcb4659662a70b2754927de434361eab0b97d5c0aced4"}
Sep 17 22:26:36.789270 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.789229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerStarted","Data":"24869248aa733c89ff5f2efea1be03a148a78cf704b626eba63ebcf23c53e779"}
Sep 17 22:26:36.789875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.789339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerStarted","Data":"3d392a8eb55a63b10e9b409a2a6e4904233565aca20edada8650b70e28aedc25"}
Sep 17 22:26:36.789875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.789566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:36.789875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.789709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:36.791399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.791372    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:26:36.791771 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.791745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerStarted","Data":"8e457f6b334f1026744d45f137bc4df1fec3e205701572920e7dfbf890aa985e"}
Sep 17 22:26:36.791866 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.791808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerStarted","Data":"17414277ea9e0a1c3405d30dc89c4c8ac98af39bcf7c036fb9d2c4d1aeef9cc1"}
Sep 17 22:26:36.792105 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.792087    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:36.792199 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.792141    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:36.793277 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.793251    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:26:36.794144 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.794122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerStarted","Data":"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"}
Sep 17 22:26:36.794231 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.794167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerStarted","Data":"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"}
Sep 17 22:26:36.794420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.794397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:36.794522 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.794456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:36.795416 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.795393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:26:36.808723 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.808673    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podStartSLOduration=3.030826964 podStartE2EDuration="6.808657221s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777830257s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:36.806603884 +0000 UTC m=+1701.474666649" watchObservedRunningTime="2026-09-17 22:26:36.808657221 +0000 UTC m=+1701.476719970"
Sep 17 22:26:36.824467 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.824403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podStartSLOduration=3.220052217 podStartE2EDuration="6.824381355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604329138s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:36.823807414 +0000 UTC m=+1701.491870198" watchObservedRunningTime="2026-09-17 22:26:36.824381355 +0000 UTC m=+1701.492444103"
Sep 17 22:26:36.851721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:36.851517    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podStartSLOduration=3.873755723 podStartE2EDuration="7.851498088s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977742365s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:36.850739596 +0000 UTC m=+1701.518802369" watchObservedRunningTime="2026-09-17 22:26:36.851498088 +0000 UTC m=+1701.519560838"
Sep 17 22:26:37.798646 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:37.798603    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:26:37.798646 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:37.798627    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:26:37.799168 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:37.798690    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:26:42.803567 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.803514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:26:42.804198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.804163    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:26:42.804611 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.804590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:26:42.804757 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.804739    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:26:42.805054 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.805032    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:26:42.805437 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:42.805410    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:26:52.804197 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:52.804154    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:26:52.805366 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:52.805340    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:26:52.805429 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:26:52.805340    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:02.804078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:02.804033    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:27:02.805324 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:02.805296    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:27:02.805324 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:02.805309    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:03.672517 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:03.672485    2582 scope.go:117] "RemoveContainer" containerID="dc97c9a38ab2b916007bef328adef25388d68304a14a49b4a2e41347747e5400"
Sep 17 22:27:12.804273 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:12.804230    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:27:12.805455 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:12.805425    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:27:12.805608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:12.805432    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:22.804030 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:22.803980    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:27:22.805250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:22.805219    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:27:22.805370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:22.805326    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:32.804935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:32.804893    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:27:32.805325 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:32.805040    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:27:32.805367 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:32.805326    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:42.804773 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:42.804736    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:27:42.805256 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:42.804972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:27:42.806172 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:42.806150    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:44.468475 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:44.468428    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:27:52.805365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:52.805336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:27:52.806139 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:52.806122    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:27:54.469789 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:27:54.469754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:28:00.011561 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.011467    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:28:00.011956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.011729    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" containerID="cri-o://17414277ea9e0a1c3405d30dc89c4c8ac98af39bcf7c036fb9d2c4d1aeef9cc1" gracePeriod=30
Sep 17 22:28:00.011956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.011772    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kube-rbac-proxy" containerID="cri-o://8e457f6b334f1026744d45f137bc4df1fec3e205701572920e7dfbf890aa985e" gracePeriod=30
Sep 17 22:28:00.952534 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.952492    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:28:00.952779 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.952752    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" containerID="cri-o://3d392a8eb55a63b10e9b409a2a6e4904233565aca20edada8650b70e28aedc25" gracePeriod=30
Sep 17 22:28:00.952870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:00.952787    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kube-rbac-proxy" containerID="cri-o://24869248aa733c89ff5f2efea1be03a148a78cf704b626eba63ebcf23c53e779" gracePeriod=30
Sep 17 22:28:01.128390 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:01.128356    2582 generic.go:356] "Generic (PLEG): container finished" podID="b54fa823-9293-4b74-a623-89904d8909a8" containerID="8e457f6b334f1026744d45f137bc4df1fec3e205701572920e7dfbf890aa985e" exitCode=2
Sep 17 22:28:01.128833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:01.128434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerDied","Data":"8e457f6b334f1026744d45f137bc4df1fec3e205701572920e7dfbf890aa985e"}
Sep 17 22:28:01.130613 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:01.130586    2582 generic.go:356] "Generic (PLEG): container finished" podID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerID="24869248aa733c89ff5f2efea1be03a148a78cf704b626eba63ebcf23c53e779" exitCode=2
Sep 17 22:28:01.130733 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:01.130657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerDied","Data":"24869248aa733c89ff5f2efea1be03a148a78cf704b626eba63ebcf23c53e779"}
Sep 17 22:28:02.799145 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:02.799098    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 22:28:02.799145 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:02.799121    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 22:28:02.804445 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:02.804409    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:28:02.805704 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:02.805673    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:28:03.691370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:03.691337    2582 scope.go:117] "RemoveContainer" containerID="259d00788be6dd52d51057c9d04b8828b331bfdd1d377f34df278e7b1181da1b"
Sep 17 22:28:03.699424 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:03.699404    2582 scope.go:117] "RemoveContainer" containerID="9d9f92388e3c461803d79db30c751f5f76dbcd8b97e77f1c2340585f67e183c2"
Sep 17 22:28:05.152255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.152222    2582 generic.go:356] "Generic (PLEG): container finished" podID="b54fa823-9293-4b74-a623-89904d8909a8" containerID="17414277ea9e0a1c3405d30dc89c4c8ac98af39bcf7c036fb9d2c4d1aeef9cc1" exitCode=0
Sep 17 22:28:05.152662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.152293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerDied","Data":"17414277ea9e0a1c3405d30dc89c4c8ac98af39bcf7c036fb9d2c4d1aeef9cc1"}
Sep 17 22:28:05.189808 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.189769    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:28:05.193710 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.193683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.196660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.196640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 22:28:05.196660 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.196654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 22:28:05.203071 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.202992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:28:05.266241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.266213    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:28:05.317469 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") pod \"b54fa823-9293-4b74-a623-89904d8909a8\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") "
Sep 17 22:28:05.317678 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location\") pod \"b54fa823-9293-4b74-a623-89904d8909a8\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") "
Sep 17 22:28:05.317678 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d56jd\" (UniqueName: \"kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd\") pod \"b54fa823-9293-4b74-a623-89904d8909a8\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") "
Sep 17 22:28:05.317678 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert\") pod \"b54fa823-9293-4b74-a623-89904d8909a8\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") "
Sep 17 22:28:05.317678 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config\") pod \"b54fa823-9293-4b74-a623-89904d8909a8\" (UID: \"b54fa823-9293-4b74-a623-89904d8909a8\") "
Sep 17 22:28:05.317916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.317916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27kv\" (UniqueName: \"kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.317916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.317916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.317916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317876    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b54fa823-9293-4b74-a623-89904d8909a8" (UID: "b54fa823-9293-4b74-a623-89904d8909a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.317986    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b54fa823-9293-4b74-a623-89904d8909a8" (UID: "b54fa823-9293-4b74-a623-89904d8909a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.318090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.318101    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b54fa823-9293-4b74-a623-89904d8909a8" (UID: "b54fa823-9293-4b74-a623-89904d8909a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.318181    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.318200    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54fa823-9293-4b74-a623-89904d8909a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.318281 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.318215    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54fa823-9293-4b74-a623-89904d8909a8-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.319853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.319831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd" (OuterVolumeSpecName: "kube-api-access-d56jd") pod "b54fa823-9293-4b74-a623-89904d8909a8" (UID: "b54fa823-9293-4b74-a623-89904d8909a8"). InnerVolumeSpecName "kube-api-access-d56jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:28:05.319853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.319842    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b54fa823-9293-4b74-a623-89904d8909a8" (UID: "b54fa823-9293-4b74-a623-89904d8909a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:28:05.418746 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.418702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.418924 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.418755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.418924 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.418875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27kv\" (UniqueName: \"kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.419024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.418944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.419024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.418984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.419128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.419055    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56jd\" (UniqueName: \"kubernetes.io/projected/b54fa823-9293-4b74-a623-89904d8909a8-kube-api-access-d56jd\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.419128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.419076    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54fa823-9293-4b74-a623-89904d8909a8-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.419223 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.419177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.419519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.419499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.419629 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.419591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.421477 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.421460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.429194 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.429172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27kv\" (UniqueName: \"kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv\") pod \"isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.506630 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.506592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:05.782523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:05.782431    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:28:05.785846 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:28:05.785817    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1669e342_7c68_4952_a074_9da124ff8356.slice/crio-3a3e000440642888e62a0a246d8607412fafb21b6aeb01b46df5b36b7edfe83d WatchSource:0}: Error finding container 3a3e000440642888e62a0a246d8607412fafb21b6aeb01b46df5b36b7edfe83d: Status 404 returned error can't find the container with id 3a3e000440642888e62a0a246d8607412fafb21b6aeb01b46df5b36b7edfe83d
Sep 17 22:28:06.160250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.160218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99" event={"ID":"b54fa823-9293-4b74-a623-89904d8909a8","Type":"ContainerDied","Data":"4c9d47d1e70b4fcd936988ed10cce99387b8c13efc9e5552fcd49521482b161b"}
Sep 17 22:28:06.160707 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.160291    2582 scope.go:117] "RemoveContainer" containerID="8e457f6b334f1026744d45f137bc4df1fec3e205701572920e7dfbf890aa985e"
Sep 17 22:28:06.160707 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.160227    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"
Sep 17 22:28:06.162174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.162147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerStarted","Data":"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"}
Sep 17 22:28:06.162312 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.162218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerStarted","Data":"3a3e000440642888e62a0a246d8607412fafb21b6aeb01b46df5b36b7edfe83d"}
Sep 17 22:28:06.173855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.173821    2582 generic.go:356] "Generic (PLEG): container finished" podID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerID="3d392a8eb55a63b10e9b409a2a6e4904233565aca20edada8650b70e28aedc25" exitCode=0
Sep 17 22:28:06.173992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.173902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerDied","Data":"3d392a8eb55a63b10e9b409a2a6e4904233565aca20edada8650b70e28aedc25"}
Sep 17 22:28:06.175860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.175842    2582 scope.go:117] "RemoveContainer" containerID="17414277ea9e0a1c3405d30dc89c4c8ac98af39bcf7c036fb9d2c4d1aeef9cc1"
Sep 17 22:28:06.186625 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.186592    2582 scope.go:117] "RemoveContainer" containerID="2e34f50824b92dec1ad5d6a2590aa29a970271f5884d1d9f138dfef52d9654ac"
Sep 17 22:28:06.198011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.197988    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:28:06.203860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.203830    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-555d868545-2lh99"]
Sep 17 22:28:06.213796 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.213773    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:28:06.325498 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325467    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config\") pod \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") "
Sep 17 22:28:06.325702 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf9bp\" (UniqueName: \"kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp\") pod \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") "
Sep 17 22:28:06.325702 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325576    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location\") pod \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") "
Sep 17 22:28:06.325702 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls\") pod \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") "
Sep 17 22:28:06.325880 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert\") pod \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\" (UID: \"06b51a82-68a2-4ad9-aefa-dbc71f6f2160\") "
Sep 17 22:28:06.325981 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.325951    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06b51a82-68a2-4ad9-aefa-dbc71f6f2160" (UID: "06b51a82-68a2-4ad9-aefa-dbc71f6f2160"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:06.326119 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.326090    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06b51a82-68a2-4ad9-aefa-dbc71f6f2160" (UID: "06b51a82-68a2-4ad9-aefa-dbc71f6f2160"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:28:06.326189 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.326173    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06b51a82-68a2-4ad9-aefa-dbc71f6f2160" (UID: "06b51a82-68a2-4ad9-aefa-dbc71f6f2160"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:28:06.327810 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.327785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06b51a82-68a2-4ad9-aefa-dbc71f6f2160" (UID: "06b51a82-68a2-4ad9-aefa-dbc71f6f2160"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:28:06.327917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.327823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp" (OuterVolumeSpecName: "kube-api-access-bf9bp") pod "06b51a82-68a2-4ad9-aefa-dbc71f6f2160" (UID: "06b51a82-68a2-4ad9-aefa-dbc71f6f2160"). InnerVolumeSpecName "kube-api-access-bf9bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:28:06.426494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.426450    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9bp\" (UniqueName: \"kubernetes.io/projected/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-api-access-bf9bp\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:06.426494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.426485    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:06.426494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.426495    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:06.426494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.426505    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:06.426788 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:06.426515    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06b51a82-68a2-4ad9-aefa-dbc71f6f2160-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:28:07.180333 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.180291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw" event={"ID":"06b51a82-68a2-4ad9-aefa-dbc71f6f2160","Type":"ContainerDied","Data":"ece1e38a2cef0d963964a45c3413b157124237d5498d83762d7463b1d85b68f0"}
Sep 17 22:28:07.180333 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.180330    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"
Sep 17 22:28:07.180882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.180378    2582 scope.go:117] "RemoveContainer" containerID="24869248aa733c89ff5f2efea1be03a148a78cf704b626eba63ebcf23c53e779"
Sep 17 22:28:07.188675 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.188631    2582 scope.go:117] "RemoveContainer" containerID="3d392a8eb55a63b10e9b409a2a6e4904233565aca20edada8650b70e28aedc25"
Sep 17 22:28:07.196288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.196268    2582 scope.go:117] "RemoveContainer" containerID="e636640a9989c4d910abcb4659662a70b2754927de434361eab0b97d5c0aced4"
Sep 17 22:28:07.202645 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.202615    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:28:07.208089 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.208064    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-qtxgw"]
Sep 17 22:28:07.472483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.472396    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" path="/var/lib/kubelet/pods/06b51a82-68a2-4ad9-aefa-dbc71f6f2160/volumes"
Sep 17 22:28:07.472920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:07.472907    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b54fa823-9293-4b74-a623-89904d8909a8" path="/var/lib/kubelet/pods/b54fa823-9293-4b74-a623-89904d8909a8/volumes"
Sep 17 22:28:10.193905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:10.193872    2582 generic.go:356] "Generic (PLEG): container finished" podID="1669e342-7c68-4952-a074-9da124ff8356" containerID="0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237" exitCode=0
Sep 17 22:28:10.194291 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:10.193937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerDied","Data":"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"}
Sep 17 22:28:11.202101 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.202057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerStarted","Data":"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"}
Sep 17 22:28:11.202573 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.202132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerStarted","Data":"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"}
Sep 17 22:28:11.202573 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.202464    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:11.202767 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.202670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:11.203989 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.203954    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:11.221313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:11.221257    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podStartSLOduration=2.344211089 podStartE2EDuration="6.221239311s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877028222s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:28:11.219819276 +0000 UTC m=+1795.887882023" watchObservedRunningTime="2026-09-17 22:28:11.221239311 +0000 UTC m=+1795.889302057"
Sep 17 22:28:12.206018 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:12.205977    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:17.210516 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:17.210487    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:28:17.211155 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:17.211095    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:27.211477 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:27.211437    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:37.211055 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:37.211012    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:47.211303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:47.211260    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:28:57.211061 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:28:57.211016    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:29:07.211016 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:07.210970    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:29:17.211386 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:17.211346    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:29:23.468299 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:23.468258    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 22:29:31.211845 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:31.211759    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:29:31.212260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:31.212012    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" containerID="cri-o://e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340" gracePeriod=30
Sep 17 22:29:31.212260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:31.212050    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kube-rbac-proxy" containerID="cri-o://ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e" gracePeriod=30
Sep 17 22:29:31.498131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:31.498042    2582 generic.go:356] "Generic (PLEG): container finished" podID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerID="ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e" exitCode=2
Sep 17 22:29:31.498131 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:31.498090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerDied","Data":"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"}
Sep 17 22:29:32.799755 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:32.799698    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 22:29:33.472223 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:33.472192    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:29:34.469246 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:34.469202    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 22:29:35.359524 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:35.359489    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:29:35.359819 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:35.359790    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container" containerID="cri-o://81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e" gracePeriod=30
Sep 17 22:29:35.359948 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:35.359806    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kube-rbac-proxy" containerID="cri-o://793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461" gracePeriod=30
Sep 17 22:29:35.515037 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:35.515001    2582 generic.go:356] "Generic (PLEG): container finished" podID="1669e342-7c68-4952-a074-9da124ff8356" containerID="793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461" exitCode=2
Sep 17 22:29:35.515435 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:35.515075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerDied","Data":"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"}
Sep 17 22:29:36.393741 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.393708    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:29:36.394072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394055    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="storage-initializer"
Sep 17 22:29:36.394072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394076    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="storage-initializer"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394086    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394094    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394103    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394112    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394125    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="storage-initializer"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394133    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="storage-initializer"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394159    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394167    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394177    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container"
Sep 17 22:29:36.394229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394185    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container"
Sep 17 22:29:36.394766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394274    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394286    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kube-rbac-proxy"
Sep 17 22:29:36.394766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394295    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b54fa823-9293-4b74-a623-89904d8909a8" containerName="kserve-container"
Sep 17 22:29:36.394766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.394305    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b51a82-68a2-4ad9-aefa-dbc71f6f2160" containerName="kserve-container"
Sep 17 22:29:36.397629 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.397605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.400921 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.400891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:29:36.401265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.400896    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 22:29:36.409857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.409082    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:29:36.465130 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.465106    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:29:36.498186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location\") pod \"35a60fae-58ba-4249-b4f8-8d16fbb01725\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") "
Sep 17 22:29:36.498186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498112    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5268\" (UniqueName: \"kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268\") pod \"35a60fae-58ba-4249-b4f8-8d16fbb01725\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") "
Sep 17 22:29:36.498186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config\") pod \"35a60fae-58ba-4249-b4f8-8d16fbb01725\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") "
Sep 17 22:29:36.498186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498176    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert\") pod \"35a60fae-58ba-4249-b4f8-8d16fbb01725\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") "
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls\") pod \"35a60fae-58ba-4249-b4f8-8d16fbb01725\" (UID: \"35a60fae-58ba-4249-b4f8-8d16fbb01725\") "
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfdw\" (UniqueName: \"kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35a60fae-58ba-4249-b4f8-8d16fbb01725" (UID: "35a60fae-58ba-4249-b4f8-8d16fbb01725"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:36.498521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.498822 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.498822 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35a60fae-58ba-4249-b4f8-8d16fbb01725" (UID: "35a60fae-58ba-4249-b4f8-8d16fbb01725"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:36.498822 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498686    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a60fae-58ba-4249-b4f8-8d16fbb01725-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.498822 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.498749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35a60fae-58ba-4249-b4f8-8d16fbb01725" (UID: "35a60fae-58ba-4249-b4f8-8d16fbb01725"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:36.500813 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.500783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268" (OuterVolumeSpecName: "kube-api-access-k5268") pod "35a60fae-58ba-4249-b4f8-8d16fbb01725" (UID: "35a60fae-58ba-4249-b4f8-8d16fbb01725"). InnerVolumeSpecName "kube-api-access-k5268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:29:36.501018 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.500997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35a60fae-58ba-4249-b4f8-8d16fbb01725" (UID: "35a60fae-58ba-4249-b4f8-8d16fbb01725"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:29:36.521621 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.521586    2582 generic.go:356] "Generic (PLEG): container finished" podID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerID="e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340" exitCode=0
Sep 17 22:29:36.521991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.521632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerDied","Data":"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"}
Sep 17 22:29:36.521991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.521658    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"
Sep 17 22:29:36.521991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.521694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k" event={"ID":"35a60fae-58ba-4249-b4f8-8d16fbb01725","Type":"ContainerDied","Data":"5397db14d2036fc7530125629cd1843b89fb2c829830386e86fcfbd9fe25d614"}
Sep 17 22:29:36.521991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.521732    2582 scope.go:117] "RemoveContainer" containerID="ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"
Sep 17 22:29:36.531259 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.531236    2582 scope.go:117] "RemoveContainer" containerID="e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"
Sep 17 22:29:36.539768 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.539749    2582 scope.go:117] "RemoveContainer" containerID="b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"
Sep 17 22:29:36.545053 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.545025    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:29:36.547780 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.547759    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-82n2k"]
Sep 17 22:29:36.548399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.548387    2582 scope.go:117] "RemoveContainer" containerID="ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"
Sep 17 22:29:36.548716 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:36.548697    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e\": container with ID starting with ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e not found: ID does not exist" containerID="ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"
Sep 17 22:29:36.548761 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.548725    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e"} err="failed to get container status \"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e\": rpc error: code = NotFound desc = could not find container \"ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e\": container with ID starting with ba072ceae6b94a5431613acecef27f605199a2699362cba52dd587bd25fbab3e not found: ID does not exist"
Sep 17 22:29:36.548761 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.548743    2582 scope.go:117] "RemoveContainer" containerID="e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"
Sep 17 22:29:36.549017 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:36.548996    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340\": container with ID starting with e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340 not found: ID does not exist" containerID="e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"
Sep 17 22:29:36.549134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.549024    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340"} err="failed to get container status \"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340\": rpc error: code = NotFound desc = could not find container \"e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340\": container with ID starting with e4f0547dfc4ed58cf0bb73999b6f8fe9bd9cecf0091796bb5964ba6943163340 not found: ID does not exist"
Sep 17 22:29:36.549134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.549045    2582 scope.go:117] "RemoveContainer" containerID="b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"
Sep 17 22:29:36.549316 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:36.549296    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8\": container with ID starting with b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8 not found: ID does not exist" containerID="b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"
Sep 17 22:29:36.549374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.549321    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8"} err="failed to get container status \"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8\": rpc error: code = NotFound desc = could not find container \"b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8\": container with ID starting with b4cfafe0643343c9a39b5091d6d90d9a3381c67d093ccbd186302aedc75316d8 not found: ID does not exist"
Sep 17 22:29:36.599870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.599831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.599870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.599873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600126 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.599909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600126 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600232 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfdw\" (UniqueName: \"kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600232 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600221    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35a60fae-58ba-4249-b4f8-8d16fbb01725-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.600347 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600240    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5268\" (UniqueName: \"kubernetes.io/projected/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-api-access-k5268\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.600347 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600255    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.600347 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600270    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35a60fae-58ba-4249-b4f8-8d16fbb01725-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.600503 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600754 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.600812 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.600735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.602498 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.602477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.608524 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.608499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfdw\" (UniqueName: \"kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw\") pod \"isvc-canary-rollback-predictor-76cbc5464-zr8qv\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.711532 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.711490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:36.989119 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:36.989083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:29:36.992241 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:29:36.992216    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a78996_261a_452b_8821_8b34cfb45854.slice/crio-47b32e1d9c7ca1feeb29e16c94fa2e5ecc47a321eb18c7325fcb3580fc7905cc WatchSource:0}: Error finding container 47b32e1d9c7ca1feeb29e16c94fa2e5ecc47a321eb18c7325fcb3580fc7905cc: Status 404 returned error can't find the container with id 47b32e1d9c7ca1feeb29e16c94fa2e5ecc47a321eb18c7325fcb3580fc7905cc
Sep 17 22:29:37.207243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:37.207191    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 22:29:37.473747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:37.473707    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" path="/var/lib/kubelet/pods/35a60fae-58ba-4249-b4f8-8d16fbb01725/volumes"
Sep 17 22:29:37.527133 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:37.527096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerStarted","Data":"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"}
Sep 17 22:29:37.527618 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:37.527186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerStarted","Data":"47b32e1d9c7ca1feeb29e16c94fa2e5ecc47a321eb18c7325fcb3580fc7905cc"}
Sep 17 22:29:40.504428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.504401    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.527778    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528151    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kube-rbac-proxy"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528170    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kube-rbac-proxy"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528182    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kube-rbac-proxy"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528190    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kube-rbac-proxy"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528205    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="storage-initializer"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528213    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="storage-initializer"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528241    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528249    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528256    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="storage-initializer"
Sep 17 22:29:40.528300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="storage-initializer"
Sep 17 22:29:40.529519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528339    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kube-rbac-proxy"
Sep 17 22:29:40.529519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528351    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kserve-container"
Sep 17 22:29:40.529519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528361    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1669e342-7c68-4952-a074-9da124ff8356" containerName="kube-rbac-proxy"
Sep 17 22:29:40.529519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.528370    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a60fae-58ba-4249-b4f8-8d16fbb01725" containerName="kserve-container"
Sep 17 22:29:40.531639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.531615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.534430 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.534406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 22:29:40.534527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.534409    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 22:29:40.541414 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.541384    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:29:40.542243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.542214    2582 generic.go:356] "Generic (PLEG): container finished" podID="1669e342-7c68-4952-a074-9da124ff8356" containerID="81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e" exitCode=0
Sep 17 22:29:40.542344 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.542326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerDied","Data":"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"}
Sep 17 22:29:40.542387 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.542378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn" event={"ID":"1669e342-7c68-4952-a074-9da124ff8356","Type":"ContainerDied","Data":"3a3e000440642888e62a0a246d8607412fafb21b6aeb01b46df5b36b7edfe83d"}
Sep 17 22:29:40.542434 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.542415    2582 scope.go:117] "RemoveContainer" containerID="793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"
Sep 17 22:29:40.542510 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.542492    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"
Sep 17 22:29:40.551808 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.551785    2582 scope.go:117] "RemoveContainer" containerID="81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"
Sep 17 22:29:40.560128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.560106    2582 scope.go:117] "RemoveContainer" containerID="0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"
Sep 17 22:29:40.568155 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.568137    2582 scope.go:117] "RemoveContainer" containerID="793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"
Sep 17 22:29:40.568451 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:40.568430    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461\": container with ID starting with 793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461 not found: ID does not exist" containerID="793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"
Sep 17 22:29:40.568513 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.568459    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461"} err="failed to get container status \"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461\": rpc error: code = NotFound desc = could not find container \"793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461\": container with ID starting with 793b3a673bcfc03e66b3c595f062559d4028abeea90eafda0dcd76f62cb1b461 not found: ID does not exist"
Sep 17 22:29:40.568513 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.568476    2582 scope.go:117] "RemoveContainer" containerID="81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"
Sep 17 22:29:40.568767 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:40.568747    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e\": container with ID starting with 81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e not found: ID does not exist" containerID="81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"
Sep 17 22:29:40.568814 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.568777    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e"} err="failed to get container status \"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e\": rpc error: code = NotFound desc = could not find container \"81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e\": container with ID starting with 81f28953dcdb8291d4d2de961a368349eb9fa832083774497292a814ecbe650e not found: ID does not exist"
Sep 17 22:29:40.568814 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.568795    2582 scope.go:117] "RemoveContainer" containerID="0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"
Sep 17 22:29:40.569058 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:40.569035    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237\": container with ID starting with 0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237 not found: ID does not exist" containerID="0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"
Sep 17 22:29:40.569109 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.569064    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237"} err="failed to get container status \"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237\": rpc error: code = NotFound desc = could not find container \"0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237\": container with ID starting with 0ee5801c506bf19a95902d402e6d904f066765dde6715c0f96f29855f8d30237 not found: ID does not exist"
Sep 17 22:29:40.628607 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config\") pod \"1669e342-7c68-4952-a074-9da124ff8356\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") "
Sep 17 22:29:40.628736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r27kv\" (UniqueName: \"kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv\") pod \"1669e342-7c68-4952-a074-9da124ff8356\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") "
Sep 17 22:29:40.628736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls\") pod \"1669e342-7c68-4952-a074-9da124ff8356\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") "
Sep 17 22:29:40.628736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert\") pod \"1669e342-7c68-4952-a074-9da124ff8356\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") "
Sep 17 22:29:40.628736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location\") pod \"1669e342-7c68-4952-a074-9da124ff8356\" (UID: \"1669e342-7c68-4952-a074-9da124ff8356\") "
Sep 17 22:29:40.628957 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.628957 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.628957 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.628935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1669e342-7c68-4952-a074-9da124ff8356" (UID: "1669e342-7c68-4952-a074-9da124ff8356"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:40.629116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1669e342-7c68-4952-a074-9da124ff8356" (UID: "1669e342-7c68-4952-a074-9da124ff8356"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:40.629116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc8f\" (UniqueName: \"kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.629116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.629220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.629220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629128    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1669e342-7c68-4952-a074-9da124ff8356" (UID: "1669e342-7c68-4952-a074-9da124ff8356"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:40.629220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629178    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:40.629220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.629191    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1669e342-7c68-4952-a074-9da124ff8356-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:40.630828 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.630809    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1669e342-7c68-4952-a074-9da124ff8356" (UID: "1669e342-7c68-4952-a074-9da124ff8356"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:29:40.631162 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.631144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv" (OuterVolumeSpecName: "kube-api-access-r27kv") pod "1669e342-7c68-4952-a074-9da124ff8356" (UID: "1669e342-7c68-4952-a074-9da124ff8356"). InnerVolumeSpecName "kube-api-access-r27kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:29:40.730398 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.730569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.730569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.730569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc8f\" (UniqueName: \"kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.730569 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:40.730527    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 22:29:40.730569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.730855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730585    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1669e342-7c68-4952-a074-9da124ff8356-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:40.730855 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:29:40.730611    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls podName:68f17cd8-a7c2-4d85-b6c9-e2af334a04c2 nodeName:}" failed. No retries permitted until 2026-09-17 22:29:41.230588708 +0000 UTC m=+1885.898651449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-b9psg" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 22:29:40.730855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730631    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1669e342-7c68-4952-a074-9da124ff8356-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:40.730855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730649    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27kv\" (UniqueName: \"kubernetes.io/projected/1669e342-7c68-4952-a074-9da124ff8356-kube-api-access-r27kv\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:29:40.731071 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.730982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.731305 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.731279    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.731380 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.731288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.739943 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.739916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc8f\" (UniqueName: \"kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:40.865621 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.865586    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:29:40.869105 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:40.869067    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-855c67d848-2d2dn"]
Sep 17 22:29:41.235100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.235056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:41.237517 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.237483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-b9psg\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:41.444658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.444620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:41.471917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.471883    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1669e342-7c68-4952-a074-9da124ff8356" path="/var/lib/kubelet/pods/1669e342-7c68-4952-a074-9da124ff8356/volumes"
Sep 17 22:29:41.549345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.549313    2582 generic.go:356] "Generic (PLEG): container finished" podID="e6a78996-261a-452b-8821-8b34cfb45854" containerID="ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af" exitCode=0
Sep 17 22:29:41.549748 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.549375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerDied","Data":"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"}
Sep 17 22:29:41.716128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:41.716091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:29:41.718226 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:29:41.718200    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f17cd8_a7c2_4d85_b6c9_e2af334a04c2.slice/crio-3301d9d34f693264620eabb327f48e19e73412436e8100dfab952bf5fead607f WatchSource:0}: Error finding container 3301d9d34f693264620eabb327f48e19e73412436e8100dfab952bf5fead607f: Status 404 returned error can't find the container with id 3301d9d34f693264620eabb327f48e19e73412436e8100dfab952bf5fead607f
Sep 17 22:29:42.555257 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.555213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerStarted","Data":"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"}
Sep 17 22:29:42.555727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.555327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerStarted","Data":"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"}
Sep 17 22:29:42.555727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.555642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:42.555727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.555691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:42.556877 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.556847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerStarted","Data":"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"}
Sep 17 22:29:42.556974 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.556915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerStarted","Data":"3301d9d34f693264620eabb327f48e19e73412436e8100dfab952bf5fead607f"}
Sep 17 22:29:42.557157 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.557134    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:29:42.577040 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:42.576992    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podStartSLOduration=2.749917902 podStartE2EDuration="6.576978874s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827060972s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:29:42.575845923 +0000 UTC m=+1887.243908691" watchObservedRunningTime="2026-09-17 22:29:42.576978874 +0000 UTC m=+1887.245041685"
Sep 17 22:29:43.561619 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:43.561565    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:29:46.574995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:46.574958    2582 generic.go:356] "Generic (PLEG): container finished" podID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerID="b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca" exitCode=0
Sep 17 22:29:46.574995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:46.574995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerDied","Data":"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"}
Sep 17 22:29:48.566662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:48.566631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:29:48.567276 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:48.567244    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:29:50.595836 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.595807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerStarted","Data":"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"}
Sep 17 22:29:50.596221 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.595873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerStarted","Data":"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"}
Sep 17 22:29:50.596221 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.596159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:50.596316 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.596297    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:50.597381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.597358    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:29:50.618001 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:50.617949    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podStartSLOduration=2.7470523890000003 podStartE2EDuration="10.617920493s" totalImagesPullingTime="3.653949104s" totalInitContainerRuntime="4.216919s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:29:50.615967595 +0000 UTC m=+1895.284030353" watchObservedRunningTime="2026-09-17 22:29:50.617920493 +0000 UTC m=+1895.285983291"
Sep 17 22:29:51.599714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:51.599677    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:29:56.604211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:56.604177    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:29:56.604839 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:56.604814    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:29:58.567487 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:29:58.567447    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:06.605832 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:06.605799    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:30:08.567803 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:08.567759    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:18.567851 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:18.567812    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:22.152036 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:22.152002    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:30:22.152855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:22.152246    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container" containerID="cri-o://545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4" gracePeriod=30
Sep 17 22:30:22.152855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:22.152287    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" containerID="cri-o://e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418" gracePeriod=30
Sep 17 22:30:22.714273 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:22.714233    2582 generic.go:356] "Generic (PLEG): container finished" podID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerID="e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418" exitCode=2
Sep 17 22:30:22.714456 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:22.714309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerDied","Data":"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"}
Sep 17 22:30:26.600162 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:26.600120    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:27.346925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.346892    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:30:27.350179 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.350160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.352937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.352909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 22:30:27.352937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.352915    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:30:27.360002 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.359976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:30:27.391847 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.391816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.392041 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.391864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.392041 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.391895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.392041 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.391952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dq5\" (UniqueName: \"kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.392041 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.391982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493020 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.492983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dq5\" (UniqueName: \"kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493020 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493279 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:30:27.493230    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 22:30:27.493279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493499 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:30:27.493324    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls podName:4a119c19-488f-4e32-b5de-e99d51c0b80f nodeName:}" failed. No retries permitted until 2026-09-17 22:30:27.993297689 +0000 UTC m=+1932.661360437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 22:30:27.493593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.493903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.493885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.502169 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.502147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dq5\" (UniqueName: \"kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.997027 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.996987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:27.999496 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:27.999470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:28.261211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:28.261119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:28.532942 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:28.532916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:30:28.535306 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:30:28.535279    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a119c19_488f_4e32_b5de_e99d51c0b80f.slice/crio-79531896848c34b4e34c1f1fceaaad06f464d445f242f6b25ee841a8e6be0378 WatchSource:0}: Error finding container 79531896848c34b4e34c1f1fceaaad06f464d445f242f6b25ee841a8e6be0378: Status 404 returned error can't find the container with id 79531896848c34b4e34c1f1fceaaad06f464d445f242f6b25ee841a8e6be0378
Sep 17 22:30:28.567235 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:28.567192    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:28.737925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:28.737885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerStarted","Data":"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"}
Sep 17 22:30:28.738130 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:28.737947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerStarted","Data":"79531896848c34b4e34c1f1fceaaad06f464d445f242f6b25ee841a8e6be0378"}
Sep 17 22:30:31.600326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:31.600281    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:33.760296 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:33.760262    2582 generic.go:356] "Generic (PLEG): container finished" podID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerID="66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01" exitCode=0
Sep 17 22:30:33.760715 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:33.760340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerDied","Data":"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"}
Sep 17 22:30:34.771301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.766848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerStarted","Data":"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"}
Sep 17 22:30:34.771301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.766995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerStarted","Data":"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"}
Sep 17 22:30:34.771301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.768837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:34.771301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.768862    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:30:34.771301 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.768915    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:34.787708 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:34.787654    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podStartSLOduration=3.2437715 podStartE2EDuration="7.787637616s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.543866116s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:30:34.786182679 +0000 UTC m=+1939.454245426" watchObservedRunningTime="2026-09-17 22:30:34.787637616 +0000 UTC m=+1939.455700364"
Sep 17 22:30:35.770608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:35.770568    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:30:36.600538 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:36.600491    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:36.600996 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:36.600697    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:30:38.567188 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:38.567148    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:40.775038 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:40.775007    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:40.775674 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:40.775643    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:30:41.600623 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:41.600569    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:46.600875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:46.600823    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:48.567882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:48.567839    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:50.776279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:50.776245    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:30:51.599982 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:51.599934    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:30:52.797680 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.797650    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:30:52.836838 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.836800    2582 generic.go:356] "Generic (PLEG): container finished" podID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerID="545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4" exitCode=137
Sep 17 22:30:52.837000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.836869    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"
Sep 17 22:30:52.837000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.836883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerDied","Data":"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"}
Sep 17 22:30:52.837000 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.836967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg" event={"ID":"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2","Type":"ContainerDied","Data":"3301d9d34f693264620eabb327f48e19e73412436e8100dfab952bf5fead607f"}
Sep 17 22:30:52.837100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.837010    2582 scope.go:117] "RemoveContainer" containerID="e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"
Sep 17 22:30:52.844905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.844888    2582 scope.go:117] "RemoveContainer" containerID="545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"
Sep 17 22:30:52.852105 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.852085    2582 scope.go:117] "RemoveContainer" containerID="b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"
Sep 17 22:30:52.859485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.859467    2582 scope.go:117] "RemoveContainer" containerID="e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"
Sep 17 22:30:52.859782 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:30:52.859764    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418\": container with ID starting with e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418 not found: ID does not exist" containerID="e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"
Sep 17 22:30:52.859825 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.859792    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418"} err="failed to get container status \"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418\": rpc error: code = NotFound desc = could not find container \"e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418\": container with ID starting with e080e7e4e1f27076ea25dcc6e1ea2f999eb07b45cf28bee4c79a794837831418 not found: ID does not exist"
Sep 17 22:30:52.859825 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.859808    2582 scope.go:117] "RemoveContainer" containerID="545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"
Sep 17 22:30:52.860027 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:30:52.860013    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4\": container with ID starting with 545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4 not found: ID does not exist" containerID="545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"
Sep 17 22:30:52.860069 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.860028    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4"} err="failed to get container status \"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4\": rpc error: code = NotFound desc = could not find container \"545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4\": container with ID starting with 545fe9212c14d7344ad3883f9744615340f9e525d16207684b0a6c74415555f4 not found: ID does not exist"
Sep 17 22:30:52.860069 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.860041    2582 scope.go:117] "RemoveContainer" containerID="b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"
Sep 17 22:30:52.860287 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:30:52.860264    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca\": container with ID starting with b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca not found: ID does not exist" containerID="b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"
Sep 17 22:30:52.860338 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.860296    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca"} err="failed to get container status \"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca\": rpc error: code = NotFound desc = could not find container \"b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca\": container with ID starting with b25a6f55de33ed967cc6bcf8b3019f64fc5b3d76bd9b53448758eac523d6c3ca not found: ID does not exist"
Sep 17 22:30:52.903174 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config\") pod \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") "
Sep 17 22:30:52.903332 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903183    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert\") pod \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") "
Sep 17 22:30:52.903332 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903246    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location\") pod \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") "
Sep 17 22:30:52.903332 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") pod \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") "
Sep 17 22:30:52.903332 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdc8f\" (UniqueName: \"kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f\") pod \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\" (UID: \"68f17cd8-a7c2-4d85-b6c9-e2af334a04c2\") "
Sep 17 22:30:52.903616 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:30:52.903670 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.903609    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:30:52.905462 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.905435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f" (OuterVolumeSpecName: "kube-api-access-qdc8f") pod "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2"). InnerVolumeSpecName "kube-api-access-qdc8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:30:52.905540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.905443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:30:52.914663 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:52.914630    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" (UID: "68f17cd8-a7c2-4d85-b6c9-e2af334a04c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:30:53.004245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.004163    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:30:53.004245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.004194    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:30:53.004245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.004204    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:30:53.004245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.004213    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdc8f\" (UniqueName: \"kubernetes.io/projected/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-api-access-qdc8f\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:30:53.004245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.004224    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:30:53.160166 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.160126    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:30:53.163510 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.163481    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-b9psg"]
Sep 17 22:30:53.473178 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:53.473148    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" path="/var/lib/kubelet/pods/68f17cd8-a7c2-4d85-b6c9-e2af334a04c2/volumes"
Sep 17 22:30:57.030326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.030286    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:30:57.030782 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.030610    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" containerID="cri-o://62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822" gracePeriod=30
Sep 17 22:30:57.030782 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.030661    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy" containerID="cri-o://74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5" gracePeriod=30
Sep 17 22:30:57.030923 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.030841    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:30:57.858782 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.858748    2582 generic.go:356] "Generic (PLEG): container finished" podID="e6a78996-261a-452b-8821-8b34cfb45854" containerID="74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5" exitCode=2
Sep 17 22:30:57.858954 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:57.858807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerDied","Data":"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"}
Sep 17 22:30:58.562470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:30:58.562426    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:31:03.568613 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.568584    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:31:03.689694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.689651    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmfdw\" (UniqueName: \"kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw\") pod \"e6a78996-261a-452b-8821-8b34cfb45854\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") "
Sep 17 22:31:03.689694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.689692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls\") pod \"e6a78996-261a-452b-8821-8b34cfb45854\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") "
Sep 17 22:31:03.689956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.689719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location\") pod \"e6a78996-261a-452b-8821-8b34cfb45854\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") "
Sep 17 22:31:03.689956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.689741    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert\") pod \"e6a78996-261a-452b-8821-8b34cfb45854\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") "
Sep 17 22:31:03.689956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.689884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config\") pod \"e6a78996-261a-452b-8821-8b34cfb45854\" (UID: \"e6a78996-261a-452b-8821-8b34cfb45854\") "
Sep 17 22:31:03.690130 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.690062    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6a78996-261a-452b-8821-8b34cfb45854" (UID: "e6a78996-261a-452b-8821-8b34cfb45854"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:03.690130 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.690077    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6a78996-261a-452b-8821-8b34cfb45854" (UID: "e6a78996-261a-452b-8821-8b34cfb45854"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:03.690247 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.690225    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a78996-261a-452b-8821-8b34cfb45854-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:03.690295 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.690254    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:03.690295 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.690285    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6a78996-261a-452b-8821-8b34cfb45854" (UID: "e6a78996-261a-452b-8821-8b34cfb45854"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:03.691908 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.691884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6a78996-261a-452b-8821-8b34cfb45854" (UID: "e6a78996-261a-452b-8821-8b34cfb45854"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:31:03.692028 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.691931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw" (OuterVolumeSpecName: "kube-api-access-xmfdw") pod "e6a78996-261a-452b-8821-8b34cfb45854" (UID: "e6a78996-261a-452b-8821-8b34cfb45854"). InnerVolumeSpecName "kube-api-access-xmfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:03.790971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.790926    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfdw\" (UniqueName: \"kubernetes.io/projected/e6a78996-261a-452b-8821-8b34cfb45854-kube-api-access-xmfdw\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:03.790971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.790962    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6a78996-261a-452b-8821-8b34cfb45854-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:03.790971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.790976    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6a78996-261a-452b-8821-8b34cfb45854-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:03.882084 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.882046    2582 generic.go:356] "Generic (PLEG): container finished" podID="e6a78996-261a-452b-8821-8b34cfb45854" containerID="62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822" exitCode=139
Sep 17 22:31:03.882250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.882108    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"
Sep 17 22:31:03.882250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.882129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerDied","Data":"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"}
Sep 17 22:31:03.882250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.882197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" event={"ID":"e6a78996-261a-452b-8821-8b34cfb45854","Type":"ContainerDied","Data":"47b32e1d9c7ca1feeb29e16c94fa2e5ecc47a321eb18c7325fcb3580fc7905cc"}
Sep 17 22:31:03.882250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.882228    2582 scope.go:117] "RemoveContainer" containerID="74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"
Sep 17 22:31:03.890439 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.890419    2582 scope.go:117] "RemoveContainer" containerID="62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"
Sep 17 22:31:03.897523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.897503    2582 scope.go:117] "RemoveContainer" containerID="ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"
Sep 17 22:31:03.903752 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.903729    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:31:03.905120 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.905104    2582 scope.go:117] "RemoveContainer" containerID="74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"
Sep 17 22:31:03.905480 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:03.905451    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5\": container with ID starting with 74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5 not found: ID does not exist" containerID="74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"
Sep 17 22:31:03.905733 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.905491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5"} err="failed to get container status \"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5\": rpc error: code = NotFound desc = could not find container \"74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5\": container with ID starting with 74e8e0d5074ef76177f376f3735d293a3110b08cbf121cd42ce839a055ab89e5 not found: ID does not exist"
Sep 17 22:31:03.905733 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.905519    2582 scope.go:117] "RemoveContainer" containerID="62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"
Sep 17 22:31:03.905882 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:03.905853    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822\": container with ID starting with 62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822 not found: ID does not exist" containerID="62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"
Sep 17 22:31:03.905949 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.905885    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822"} err="failed to get container status \"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822\": rpc error: code = NotFound desc = could not find container \"62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822\": container with ID starting with 62e58483dba853f7a68a9764014526df3df45ddd36f8f85976e6635bf3876822 not found: ID does not exist"
Sep 17 22:31:03.905949 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.905904    2582 scope.go:117] "RemoveContainer" containerID="ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"
Sep 17 22:31:03.906182 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:03.906154    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af\": container with ID starting with ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af not found: ID does not exist" containerID="ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"
Sep 17 22:31:03.906397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.906177    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af"} err="failed to get container status \"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af\": rpc error: code = NotFound desc = could not find container \"ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af\": container with ID starting with ebe21a26ac892914c48d670bc8893d48e2e94c444378f382c467271616cca0af not found: ID does not exist"
Sep 17 22:31:03.907381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:03.907361    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv"]
Sep 17 22:31:04.562294 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:04.562243    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-zr8qv" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": context deadline exceeded"
Sep 17 22:31:05.472803 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:05.472772    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a78996-261a-452b-8821-8b34cfb45854" path="/var/lib/kubelet/pods/e6a78996-261a-452b-8821-8b34cfb45854/volumes"
Sep 17 22:31:06.419222 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419179    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:31:06.419528 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419511    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419530    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419584    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container"
Sep 17 22:31:06.419639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419594    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container"
Sep 17 22:31:06.419639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419608    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419618    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419643    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="storage-initializer"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419651    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="storage-initializer"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419672    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="storage-initializer"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="storage-initializer"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419692    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419701    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419774    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419784    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f17cd8-a7c2-4d85-b6c9-e2af334a04c2" containerName="kserve-container"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419794    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kube-rbac-proxy"
Sep 17 22:31:06.419903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.419803    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a78996-261a-452b-8821-8b34cfb45854" containerName="kserve-container"
Sep 17 22:31:06.422777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.422754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.425396 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.425377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 22:31:06.425396 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.425386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:31:06.431326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.431301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:31:06.510684 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.510641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.511102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.510689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.511102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.510739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.511102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.510765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl7z\" (UniqueName: \"kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.511102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.510844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612124 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612124 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612415 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612415 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl7z\" (UniqueName: \"kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612415 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.612744 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.613009 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.612991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.613053 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.613030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.614659 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.614638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.621388 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.621363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl7z\" (UniqueName: \"kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z\") pod \"isvc-canary-rollback-predictor-76cbc5464-mndx4\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:06.734448 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:06.734350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:07.009407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:07.009379    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:31:07.012029 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:31:07.011996    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d588807_742c_4516_a29e_1730835a9b8d.slice/crio-583982444dd656f41e77ec49af1580a532260c7f964f7d2282e91f73154d91f9 WatchSource:0}: Error finding container 583982444dd656f41e77ec49af1580a532260c7f964f7d2282e91f73154d91f9: Status 404 returned error can't find the container with id 583982444dd656f41e77ec49af1580a532260c7f964f7d2282e91f73154d91f9
Sep 17 22:31:07.897899 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:07.897861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerStarted","Data":"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"}
Sep 17 22:31:07.898393 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:07.897935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerStarted","Data":"583982444dd656f41e77ec49af1580a532260c7f964f7d2282e91f73154d91f9"}
Sep 17 22:31:08.943597 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:08.943563    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:31:08.944074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:08.943864    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container" containerID="cri-o://e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa" gracePeriod=30
Sep 17 22:31:08.944074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:08.943909    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" containerID="cri-o://f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb" gracePeriod=30
Sep 17 22:31:09.907284 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:09.907252    2582 generic.go:356] "Generic (PLEG): container finished" podID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerID="f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb" exitCode=2
Sep 17 22:31:09.907445 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:09.907291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerDied","Data":"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"}
Sep 17 22:31:10.771853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:10.771806    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:11.915870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:11.915833    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d588807-742c-4516-a29e-1730835a9b8d" containerID="d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d" exitCode=0
Sep 17 22:31:11.916326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:11.915908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerDied","Data":"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"}
Sep 17 22:31:12.922354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.922318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerStarted","Data":"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"}
Sep 17 22:31:12.922772 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.922382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerStarted","Data":"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"}
Sep 17 22:31:12.922772 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.922673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:12.922876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.922812    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:12.924254 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.924226    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:12.941378 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:12.941329    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podStartSLOduration=3.04050791 podStartE2EDuration="6.941314346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.900806436s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:31:12.939453817 +0000 UTC m=+1977.607516584" watchObservedRunningTime="2026-09-17 22:31:12.941314346 +0000 UTC m=+1977.609377093"
Sep 17 22:31:13.926417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:13.926372    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:15.771503 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:15.771457    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:18.930909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:18.930878    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:31:18.931423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:18.931397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:20.771197 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:20.771148    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:20.771637 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:20.771293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:31:25.771639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:25.771595    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:28.931460 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:28.931418    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:30.771511 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:30.771463    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:35.770803 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:35.770752    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:31:38.931653 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:38.931605    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:39.589121 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.589097    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:31:39.675651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.675611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config\") pod \"4a119c19-488f-4e32-b5de-e99d51c0b80f\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") "
Sep 17 22:31:39.675651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.675654    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location\") pod \"4a119c19-488f-4e32-b5de-e99d51c0b80f\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") "
Sep 17 22:31:39.675905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.675695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert\") pod \"4a119c19-488f-4e32-b5de-e99d51c0b80f\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") "
Sep 17 22:31:39.675905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.675724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") pod \"4a119c19-488f-4e32-b5de-e99d51c0b80f\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") "
Sep 17 22:31:39.675905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.675776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dq5\" (UniqueName: \"kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5\") pod \"4a119c19-488f-4e32-b5de-e99d51c0b80f\" (UID: \"4a119c19-488f-4e32-b5de-e99d51c0b80f\") "
Sep 17 22:31:39.676126 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.676093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a119c19-488f-4e32-b5de-e99d51c0b80f" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:39.676126 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.676109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a119c19-488f-4e32-b5de-e99d51c0b80f" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:39.678095 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.678058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5" (OuterVolumeSpecName: "kube-api-access-24dq5") pod "4a119c19-488f-4e32-b5de-e99d51c0b80f" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f"). InnerVolumeSpecName "kube-api-access-24dq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:39.678214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.678090    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a119c19-488f-4e32-b5de-e99d51c0b80f" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:31:39.684704 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.684680    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a119c19-488f-4e32-b5de-e99d51c0b80f" (UID: "4a119c19-488f-4e32-b5de-e99d51c0b80f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:39.776875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.776775    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:39.776875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.776820    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a119c19-488f-4e32-b5de-e99d51c0b80f-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:39.776875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.776832    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a119c19-488f-4e32-b5de-e99d51c0b80f-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:39.776875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.776841    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a119c19-488f-4e32-b5de-e99d51c0b80f-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:39.776875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:39.776849    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dq5\" (UniqueName: \"kubernetes.io/projected/4a119c19-488f-4e32-b5de-e99d51c0b80f-kube-api-access-24dq5\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.023114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.023076    2582 generic.go:356] "Generic (PLEG): container finished" podID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerID="e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa" exitCode=137
Sep 17 22:31:40.023665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.023126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerDied","Data":"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"}
Sep 17 22:31:40.023665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.023138    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"
Sep 17 22:31:40.023665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.023185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn" event={"ID":"4a119c19-488f-4e32-b5de-e99d51c0b80f","Type":"ContainerDied","Data":"79531896848c34b4e34c1f1fceaaad06f464d445f242f6b25ee841a8e6be0378"}
Sep 17 22:31:40.023665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.023212    2582 scope.go:117] "RemoveContainer" containerID="f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"
Sep 17 22:31:40.031304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.031281    2582 scope.go:117] "RemoveContainer" containerID="e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"
Sep 17 22:31:40.038671 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.038655    2582 scope.go:117] "RemoveContainer" containerID="66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"
Sep 17 22:31:40.045162 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.045138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:31:40.046304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.046290    2582 scope.go:117] "RemoveContainer" containerID="f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"
Sep 17 22:31:40.046614 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:40.046592    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb\": container with ID starting with f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb not found: ID does not exist" containerID="f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"
Sep 17 22:31:40.046693 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.046624    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb"} err="failed to get container status \"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb\": rpc error: code = NotFound desc = could not find container \"f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb\": container with ID starting with f90860cc679a3026686e301abb66f7798d259e911493f382193232ac60d421eb not found: ID does not exist"
Sep 17 22:31:40.046693 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.046644    2582 scope.go:117] "RemoveContainer" containerID="e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"
Sep 17 22:31:40.046950 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:40.046924    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa\": container with ID starting with e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa not found: ID does not exist" containerID="e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"
Sep 17 22:31:40.046950 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.046956    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa"} err="failed to get container status \"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa\": rpc error: code = NotFound desc = could not find container \"e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa\": container with ID starting with e1f7419d29ef4d8f8e42a04c2d74680d8bc66201288f71a3675d1c82c7c33afa not found: ID does not exist"
Sep 17 22:31:40.047152 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.046980    2582 scope.go:117] "RemoveContainer" containerID="66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"
Sep 17 22:31:40.047328 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:31:40.047307    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01\": container with ID starting with 66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01 not found: ID does not exist" containerID="66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"
Sep 17 22:31:40.047575 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.047331    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01"} err="failed to get container status \"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01\": rpc error: code = NotFound desc = could not find container \"66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01\": container with ID starting with 66ec5a555c18787cee5d626e4f06017af6fb9923d9d855ce7a5ac459d6fa6b01 not found: ID does not exist"
Sep 17 22:31:40.048887 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:40.048869    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-h9mdn"]
Sep 17 22:31:41.472954 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:41.472926    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" path="/var/lib/kubelet/pods/4a119c19-488f-4e32-b5de-e99d51c0b80f/volumes"
Sep 17 22:31:48.931724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:48.931684    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:31:58.931367 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:31:58.931323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:32:08.932035 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:08.931988    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:32:18.931992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:18.931942    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:32:28.932395 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:28.932358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:32:36.571794 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:36.571756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:32:36.572289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:36.572016    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kube-rbac-proxy" containerID="cri-o://0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3" gracePeriod=30
Sep 17 22:32:36.572289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:36.572010    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" containerID="cri-o://96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8" gracePeriod=30
Sep 17 22:32:37.235388 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:37.235352    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d588807-742c-4516-a29e-1730835a9b8d" containerID="0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3" exitCode=2
Sep 17 22:32:37.235630 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:37.235429    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerDied","Data":"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"}
Sep 17 22:32:38.927270 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:38.927222    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:32:38.931606 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:38.931575    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:32:41.616109 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.616079    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:32:41.716329 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716243    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:32:41.716536 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716524    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716538    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716571    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716578    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kube-rbac-proxy"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716584    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kube-rbac-proxy"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716594    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="storage-initializer"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716599    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="storage-initializer"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716611    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716618    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="storage-initializer"
Sep 17 22:32:41.716636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716623    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="storage-initializer"
Sep 17 22:32:41.717079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kube-rbac-proxy"
Sep 17 22:32:41.717079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716675    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kserve-container"
Sep 17 22:32:41.717079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716682    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a119c19-488f-4e32-b5de-e99d51c0b80f" containerName="kube-rbac-proxy"
Sep 17 22:32:41.717079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.716687    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d588807-742c-4516-a29e-1730835a9b8d" containerName="kserve-container"
Sep 17 22:32:41.719603 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.719586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.722255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.722226    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 22:32:41.722255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.722244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 22:32:41.730190 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.730161    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:32:41.735989 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.735960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls\") pod \"4d588807-742c-4516-a29e-1730835a9b8d\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") "
Sep 17 22:32:41.736171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert\") pod \"4d588807-742c-4516-a29e-1730835a9b8d\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") "
Sep 17 22:32:41.736171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736054    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config\") pod \"4d588807-742c-4516-a29e-1730835a9b8d\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") "
Sep 17 22:32:41.736171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736093    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location\") pod \"4d588807-742c-4516-a29e-1730835a9b8d\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") "
Sep 17 22:32:41.736171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddl7z\" (UniqueName: \"kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z\") pod \"4d588807-742c-4516-a29e-1730835a9b8d\" (UID: \"4d588807-742c-4516-a29e-1730835a9b8d\") "
Sep 17 22:32:41.736444 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736417    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d588807-742c-4516-a29e-1730835a9b8d" (UID: "4d588807-742c-4516-a29e-1730835a9b8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:32:41.736519 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736455    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d588807-742c-4516-a29e-1730835a9b8d" (UID: "4d588807-742c-4516-a29e-1730835a9b8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:32:41.736867 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.736817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d588807-742c-4516-a29e-1730835a9b8d" (UID: "4d588807-742c-4516-a29e-1730835a9b8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:41.738777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.738750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z" (OuterVolumeSpecName: "kube-api-access-ddl7z") pod "4d588807-742c-4516-a29e-1730835a9b8d" (UID: "4d588807-742c-4516-a29e-1730835a9b8d"). InnerVolumeSpecName "kube-api-access-ddl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:32:41.738975 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.738959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d588807-742c-4516-a29e-1730835a9b8d" (UID: "4d588807-742c-4516-a29e-1730835a9b8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:32:41.749143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.749116    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:32:41.752829 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.752812    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.755369 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.755346    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 22:32:41.763436 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.763410    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:32:41.837738 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.837738 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfmz\" (UniqueName: \"kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6tp\" (UniqueName: \"kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.837984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.838033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.838029    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:32:41.838323 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.838041    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d588807-742c-4516-a29e-1730835a9b8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:32:41.838323 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.838050    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d588807-742c-4516-a29e-1730835a9b8d-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:32:41.838323 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.838059    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddl7z\" (UniqueName: \"kubernetes.io/projected/4d588807-742c-4516-a29e-1730835a9b8d-kube-api-access-ddl7z\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:32:41.838323 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.838074    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d588807-742c-4516-a29e-1730835a9b8d-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:32:41.938793 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfmz\" (UniqueName: \"kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.938988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6tp\" (UniqueName: \"kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.939374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.938994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.939374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.939374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.939374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.939374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.939725 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.939813 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.939865 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.939938 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.939920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.942082 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.942061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.942171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.942069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:41.946958 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.946936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfmz\" (UniqueName: \"kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:41.947381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:41.947365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6tp\" (UniqueName: \"kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp\") pod \"isvc-canary-stop-predictor-646f5bdbcb-ngjdt\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:42.030829 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.030736    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:42.063664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.063625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:42.254599 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.254563    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d588807-742c-4516-a29e-1730835a9b8d" containerID="96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8" exitCode=0
Sep 17 22:32:42.254769 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.254634    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"
Sep 17 22:32:42.254769 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.254648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerDied","Data":"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"}
Sep 17 22:32:42.254769 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.254702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4" event={"ID":"4d588807-742c-4516-a29e-1730835a9b8d","Type":"ContainerDied","Data":"583982444dd656f41e77ec49af1580a532260c7f964f7d2282e91f73154d91f9"}
Sep 17 22:32:42.254769 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.254728    2582 scope.go:117] "RemoveContainer" containerID="0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"
Sep 17 22:32:42.263887 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.263625    2582 scope.go:117] "RemoveContainer" containerID="96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"
Sep 17 22:32:42.271643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.271598    2582 scope.go:117] "RemoveContainer" containerID="d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"
Sep 17 22:32:42.278623 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.278593    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:32:42.279440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.279425    2582 scope.go:117] "RemoveContainer" containerID="0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"
Sep 17 22:32:42.279860 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:32:42.279835    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3\": container with ID starting with 0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3 not found: ID does not exist" containerID="0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"
Sep 17 22:32:42.279955 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.279868    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3"} err="failed to get container status \"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3\": rpc error: code = NotFound desc = could not find container \"0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3\": container with ID starting with 0fee6fe5de0f3f15d5d77c066f70c878980d86841f5d294d7e46d6157f8acea3 not found: ID does not exist"
Sep 17 22:32:42.279955 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.279887    2582 scope.go:117] "RemoveContainer" containerID="96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"
Sep 17 22:32:42.280187 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:32:42.280161    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8\": container with ID starting with 96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8 not found: ID does not exist" containerID="96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"
Sep 17 22:32:42.280248 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.280195    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8"} err="failed to get container status \"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8\": rpc error: code = NotFound desc = could not find container \"96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8\": container with ID starting with 96e01bb2d1040f4c0c052ffdddc7ccc678c8433e12354d2b061d84013c7f4fd8 not found: ID does not exist"
Sep 17 22:32:42.280248 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.280212    2582 scope.go:117] "RemoveContainer" containerID="d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"
Sep 17 22:32:42.280662 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:32:42.280565    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d\": container with ID starting with d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d not found: ID does not exist" containerID="d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"
Sep 17 22:32:42.280662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.280597    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d"} err="failed to get container status \"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d\": rpc error: code = NotFound desc = could not find container \"d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d\": container with ID starting with d8c8f61e6c08b22d183ec046ef9578c2170a73f427e4a2d6c447a0551b47cd4d not found: ID does not exist"
Sep 17 22:32:42.282157 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.282107    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-mndx4"]
Sep 17 22:32:42.308098 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.308070    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:32:42.310639 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:32:42.310609    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a09fad3_de4d_4db3_8dd1_d5c98ff11725.slice/crio-4dba0606388eac03da475bfada7f524270b1c339988848395cbbc26b22374e8d WatchSource:0}: Error finding container 4dba0606388eac03da475bfada7f524270b1c339988848395cbbc26b22374e8d: Status 404 returned error can't find the container with id 4dba0606388eac03da475bfada7f524270b1c339988848395cbbc26b22374e8d
Sep 17 22:32:42.346159 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:42.346025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:32:42.348661 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:32:42.348633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088a252d_4eef_421f_8dc9_c76232363bb8.slice/crio-e36f3dc3bcbc39dc0ba544e6fe3e3962df81302a134e05e8e5e7bd7016e60c9f WatchSource:0}: Error finding container e36f3dc3bcbc39dc0ba544e6fe3e3962df81302a134e05e8e5e7bd7016e60c9f: Status 404 returned error can't find the container with id e36f3dc3bcbc39dc0ba544e6fe3e3962df81302a134e05e8e5e7bd7016e60c9f
Sep 17 22:32:43.260441 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:43.260403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerStarted","Data":"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"}
Sep 17 22:32:43.260938 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:43.260478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerStarted","Data":"4dba0606388eac03da475bfada7f524270b1c339988848395cbbc26b22374e8d"}
Sep 17 22:32:43.262652 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:43.262625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerStarted","Data":"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"}
Sep 17 22:32:43.262799 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:43.262673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerStarted","Data":"e36f3dc3bcbc39dc0ba544e6fe3e3962df81302a134e05e8e5e7bd7016e60c9f"}
Sep 17 22:32:43.472847 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:43.472814    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d588807-742c-4516-a29e-1730835a9b8d" path="/var/lib/kubelet/pods/4d588807-742c-4516-a29e-1730835a9b8d/volumes"
Sep 17 22:32:46.276539 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:46.276499    2582 generic.go:356] "Generic (PLEG): container finished" podID="088a252d-4eef-421f-8dc9-c76232363bb8" containerID="7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c" exitCode=0
Sep 17 22:32:46.277013 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:46.276573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerDied","Data":"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"}
Sep 17 22:32:46.278280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:46.278248    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerID="b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f" exitCode=0
Sep 17 22:32:46.278366 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:46.278286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerDied","Data":"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"}
Sep 17 22:32:47.285029 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.284989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerStarted","Data":"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"}
Sep 17 22:32:47.285473 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.285068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerStarted","Data":"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"}
Sep 17 22:32:47.285473 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.285368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:47.285473 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.285424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:47.286957 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.286924    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:32:47.287293 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.287275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerStarted","Data":"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"}
Sep 17 22:32:47.287382 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.287322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerStarted","Data":"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"}
Sep 17 22:32:47.287510 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.287493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:47.287585 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.287575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:47.288565 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.288517    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:32:47.304008 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.303964    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podStartSLOduration=2.562104147 podStartE2EDuration="6.30395024s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741846093s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:32:47.302815693 +0000 UTC m=+2071.970878441" watchObservedRunningTime="2026-09-17 22:32:47.30395024 +0000 UTC m=+2071.972012987"
Sep 17 22:32:47.322557 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:47.322503    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podStartSLOduration=2.574425455 podStartE2EDuration="6.322486323s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.748060868s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:32:47.320840718 +0000 UTC m=+2071.988903483" watchObservedRunningTime="2026-09-17 22:32:47.322486323 +0000 UTC m=+2071.990549069"
Sep 17 22:32:48.291206 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:48.291164    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:32:48.291681 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:48.291266    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:32:53.295645 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:53.295615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:32:53.296149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:53.296130    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:32:53.296198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:53.296145    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:32:53.296595 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:32:53.296571    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:03.296342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:03.296304    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:03.296836 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:03.296613    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:13.296694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:13.296655    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:13.297075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:13.296654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:23.296705 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:23.296661    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:23.297097 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:23.296665    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:33.296448 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:33.296407    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:33.297003 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:33.296535    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:41.145710 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.145672    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:33:41.148747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.148728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.151278 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.151247    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 22:33:41.151422 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.151330    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 22:33:41.159624 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.159598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:33:41.203316 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.203273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.203516 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.203338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.203516 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.203395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.203516 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.203429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.203516 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.203465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcz7\" (UniqueName: \"kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcz7\" (UniqueName: \"kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.304827 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.304799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.305075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.305057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.305132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.305072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.306928 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.306909    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.312778 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.312748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcz7\" (UniqueName: \"kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7\") pod \"isvc-xgboost-predictor-7566d65bf7-v6lj9\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.459667 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.459537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:33:41.731768 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:41.731709    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:33:41.734759 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:33:41.734715    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae45340f_584f_4985_9962_a4c8f694b5c5.slice/crio-71b2a650a7654cea55e4be071bfad67e920ec49a810d6d01d20735283c091448 WatchSource:0}: Error finding container 71b2a650a7654cea55e4be071bfad67e920ec49a810d6d01d20735283c091448: Status 404 returned error can't find the container with id 71b2a650a7654cea55e4be071bfad67e920ec49a810d6d01d20735283c091448
Sep 17 22:33:42.489042 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:42.488651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerStarted","Data":"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"}
Sep 17 22:33:42.489042 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:42.488771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerStarted","Data":"71b2a650a7654cea55e4be071bfad67e920ec49a810d6d01d20735283c091448"}
Sep 17 22:33:43.297026 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:43.296991    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:43.297244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:43.296991    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:45.502530 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:45.502495    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerID="2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833" exitCode=0
Sep 17 22:33:45.502937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:45.502572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerDied","Data":"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"}
Sep 17 22:33:45.503790 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:45.503773    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:33:53.296542 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:53.296484    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:33:53.297431 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:53.297205    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:33:57.469727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:33:57.469501    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 22:34:03.296885 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:03.296781    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:34:04.586805 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:04.586767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerStarted","Data":"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"}
Sep 17 22:34:04.587186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:04.586835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerStarted","Data":"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"}
Sep 17 22:34:04.587186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:04.587031    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:34:04.607005 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:04.606916    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podStartSLOduration=1.456797672 podStartE2EDuration="23.606899669s" totalImagesPullingTime="18.583772093s" totalInitContainerRuntime="3.566329904s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:34:04.604426126 +0000 UTC m=+2149.272488894" watchObservedRunningTime="2026-09-17 22:34:04.606899669 +0000 UTC m=+2149.274962416"
Sep 17 22:34:05.590928 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:05.590895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:34:05.592116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:05.592093    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:06.594544 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:06.594505    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:07.471968 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:07.471940    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:34:11.598648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:11.598615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:34:11.599104 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:11.599077    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:13.296718 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:13.296688    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:34:21.599107 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.599065    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:21.745675 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.745633    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:34:21.745937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.745910    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" containerID="cri-o://8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b" gracePeriod=30
Sep 17 22:34:21.746021 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.745988    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kube-rbac-proxy" containerID="cri-o://a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48" gracePeriod=30
Sep 17 22:34:21.780153 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.780115    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:34:21.780414 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.780387    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container" containerID="cri-o://000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d" gracePeriod=30
Sep 17 22:34:21.780504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:21.780412    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kube-rbac-proxy" containerID="cri-o://0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904" gracePeriod=30
Sep 17 22:34:22.662874 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:22.662837    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerID="a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48" exitCode=2
Sep 17 22:34:22.663291 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:22.662902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerDied","Data":"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"}
Sep 17 22:34:22.665040 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:22.665020    2582 generic.go:356] "Generic (PLEG): container finished" podID="088a252d-4eef-421f-8dc9-c76232363bb8" containerID="0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904" exitCode=2
Sep 17 22:34:22.665140 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:22.665063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerDied","Data":"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"}
Sep 17 22:34:23.292289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:23.292251    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 22:34:23.292468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:23.292251    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 22:34:23.296591 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:23.296540    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 22:34:27.020311 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.020281    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:34:27.084761 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.084731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config\") pod \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") "
Sep 17 22:34:27.084952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.084787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert\") pod \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") "
Sep 17 22:34:27.084952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.084827    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location\") pod \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") "
Sep 17 22:34:27.084952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.084846    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpfmz\" (UniqueName: \"kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz\") pod \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") "
Sep 17 22:34:27.084952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.084897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls\") pod \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\" (UID: \"7a09fad3-de4d-4db3-8dd1-d5c98ff11725\") "
Sep 17 22:34:27.085252 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.085210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a09fad3-de4d-4db3-8dd1-d5c98ff11725" (UID: "7a09fad3-de4d-4db3-8dd1-d5c98ff11725"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:34:27.085252 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.085236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a09fad3-de4d-4db3-8dd1-d5c98ff11725" (UID: "7a09fad3-de4d-4db3-8dd1-d5c98ff11725"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:27.085252 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.085243    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a09fad3-de4d-4db3-8dd1-d5c98ff11725" (UID: "7a09fad3-de4d-4db3-8dd1-d5c98ff11725"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:27.087032 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.087003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a09fad3-de4d-4db3-8dd1-d5c98ff11725" (UID: "7a09fad3-de4d-4db3-8dd1-d5c98ff11725"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:34:27.087221 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.087207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz" (OuterVolumeSpecName: "kube-api-access-xpfmz") pod "7a09fad3-de4d-4db3-8dd1-d5c98ff11725" (UID: "7a09fad3-de4d-4db3-8dd1-d5c98ff11725"). InnerVolumeSpecName "kube-api-access-xpfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:34:27.117685 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.117658    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:34:27.186214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls\") pod \"088a252d-4eef-421f-8dc9-c76232363bb8\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") "
Sep 17 22:34:27.186402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config\") pod \"088a252d-4eef-421f-8dc9-c76232363bb8\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") "
Sep 17 22:34:27.186402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert\") pod \"088a252d-4eef-421f-8dc9-c76232363bb8\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") "
Sep 17 22:34:27.186402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186381    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location\") pod \"088a252d-4eef-421f-8dc9-c76232363bb8\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") "
Sep 17 22:34:27.186601 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms6tp\" (UniqueName: \"kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp\") pod \"088a252d-4eef-421f-8dc9-c76232363bb8\" (UID: \"088a252d-4eef-421f-8dc9-c76232363bb8\") "
Sep 17 22:34:27.186713 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186684    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.186775 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186725    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.186775 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186741    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.186775 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "088a252d-4eef-421f-8dc9-c76232363bb8" (UID: "088a252d-4eef-421f-8dc9-c76232363bb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:27.186882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186755    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfmz\" (UniqueName: \"kubernetes.io/projected/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-kube-api-access-xpfmz\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.186882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186702    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "088a252d-4eef-421f-8dc9-c76232363bb8" (UID: "088a252d-4eef-421f-8dc9-c76232363bb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:27.186882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "088a252d-4eef-421f-8dc9-c76232363bb8" (UID: "088a252d-4eef-421f-8dc9-c76232363bb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:34:27.186882 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.186800    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a09fad3-de4d-4db3-8dd1-d5c98ff11725-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.188420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.188401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "088a252d-4eef-421f-8dc9-c76232363bb8" (UID: "088a252d-4eef-421f-8dc9-c76232363bb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:34:27.188493 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.188441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp" (OuterVolumeSpecName: "kube-api-access-ms6tp") pod "088a252d-4eef-421f-8dc9-c76232363bb8" (UID: "088a252d-4eef-421f-8dc9-c76232363bb8"). InnerVolumeSpecName "kube-api-access-ms6tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:34:27.287345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.287308    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.287345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.287338    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088a252d-4eef-421f-8dc9-c76232363bb8-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.287345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.287347    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088a252d-4eef-421f-8dc9-c76232363bb8-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.287345 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.287357    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6tp\" (UniqueName: \"kubernetes.io/projected/088a252d-4eef-421f-8dc9-c76232363bb8-kube-api-access-ms6tp\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.287649 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.287367    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088a252d-4eef-421f-8dc9-c76232363bb8-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:34:27.684992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.684959    2582 generic.go:356] "Generic (PLEG): container finished" podID="088a252d-4eef-421f-8dc9-c76232363bb8" containerID="000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d" exitCode=0
Sep 17 22:34:27.685149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.685022    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"
Sep 17 22:34:27.685149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.685041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerDied","Data":"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"}
Sep 17 22:34:27.685149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.685103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt" event={"ID":"088a252d-4eef-421f-8dc9-c76232363bb8","Type":"ContainerDied","Data":"e36f3dc3bcbc39dc0ba544e6fe3e3962df81302a134e05e8e5e7bd7016e60c9f"}
Sep 17 22:34:27.685149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.685138    2582 scope.go:117] "RemoveContainer" containerID="0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"
Sep 17 22:34:27.687389 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.687347    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerID="8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b" exitCode=0
Sep 17 22:34:27.687506 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.687398    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerDied","Data":"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"}
Sep 17 22:34:27.687506 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.687429    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"
Sep 17 22:34:27.687506 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.687473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24" event={"ID":"7a09fad3-de4d-4db3-8dd1-d5c98ff11725","Type":"ContainerDied","Data":"4dba0606388eac03da475bfada7f524270b1c339988848395cbbc26b22374e8d"}
Sep 17 22:34:27.693418 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.693400    2582 scope.go:117] "RemoveContainer" containerID="000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"
Sep 17 22:34:27.700627 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.700607    2582 scope.go:117] "RemoveContainer" containerID="7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"
Sep 17 22:34:27.706104 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.706081    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:34:27.708729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.708698    2582 scope.go:117] "RemoveContainer" containerID="0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"
Sep 17 22:34:27.709072 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.709054    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904\": container with ID starting with 0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904 not found: ID does not exist" containerID="0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"
Sep 17 22:34:27.709125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709081    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904"} err="failed to get container status \"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904\": rpc error: code = NotFound desc = could not find container \"0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904\": container with ID starting with 0cf711da8722e057951f8c476dc1f833f2ae808cd37cc6b7eb87be0303667904 not found: ID does not exist"
Sep 17 22:34:27.709125 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709100    2582 scope.go:117] "RemoveContainer" containerID="000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"
Sep 17 22:34:27.709365 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.709348    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d\": container with ID starting with 000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d not found: ID does not exist" containerID="000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"
Sep 17 22:34:27.709435 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709376    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d"} err="failed to get container status \"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d\": rpc error: code = NotFound desc = could not find container \"000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d\": container with ID starting with 000d6608a73909f87302138466b485ab918c7eb4f50b8e15f84735b2b16d233d not found: ID does not exist"
Sep 17 22:34:27.709435 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709400    2582 scope.go:117] "RemoveContainer" containerID="7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"
Sep 17 22:34:27.709705 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.709688    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c\": container with ID starting with 7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c not found: ID does not exist" containerID="7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"
Sep 17 22:34:27.709765 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709711    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c"} err="failed to get container status \"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c\": rpc error: code = NotFound desc = could not find container \"7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c\": container with ID starting with 7d744c9c80e0eac7fba82a0d26a25df2085f731da48097870388b786dad35a5c not found: ID does not exist"
Sep 17 22:34:27.709765 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709724    2582 scope.go:117] "RemoveContainer" containerID="a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"
Sep 17 22:34:27.709982 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.709961    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-ngjdt"]
Sep 17 22:34:27.717021 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.717003    2582 scope.go:117] "RemoveContainer" containerID="8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"
Sep 17 22:34:27.720390 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.720367    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:34:27.724406 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.724380    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-hcq24"]
Sep 17 22:34:27.724682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.724665    2582 scope.go:117] "RemoveContainer" containerID="b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"
Sep 17 22:34:27.731760 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.731740    2582 scope.go:117] "RemoveContainer" containerID="a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"
Sep 17 22:34:27.732043 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.732025    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48\": container with ID starting with a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48 not found: ID does not exist" containerID="a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"
Sep 17 22:34:27.732090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.732052    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48"} err="failed to get container status \"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48\": rpc error: code = NotFound desc = could not find container \"a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48\": container with ID starting with a986be8db348ba309e1bba46d2f6d4b1a325996b43a0dd5a3b81698a2b2b9a48 not found: ID does not exist"
Sep 17 22:34:27.732090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.732070    2582 scope.go:117] "RemoveContainer" containerID="8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"
Sep 17 22:34:27.732327 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.732310    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b\": container with ID starting with 8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b not found: ID does not exist" containerID="8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"
Sep 17 22:34:27.732374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.732330    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b"} err="failed to get container status \"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b\": rpc error: code = NotFound desc = could not find container \"8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b\": container with ID starting with 8a08069ffe4649f942de7426dd5004965f4d0e94c48291d1c965f464a99cb84b not found: ID does not exist"
Sep 17 22:34:27.732374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.732343    2582 scope.go:117] "RemoveContainer" containerID="b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"
Sep 17 22:34:27.732605 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:34:27.732582    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f\": container with ID starting with b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f not found: ID does not exist" containerID="b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"
Sep 17 22:34:27.732692 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:27.732612    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f"} err="failed to get container status \"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f\": rpc error: code = NotFound desc = could not find container \"b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f\": container with ID starting with b1d0a6a1f0a996a2a999c5dd14e1208e938054c7fc26b9741d933adb80b7a76f not found: ID does not exist"
Sep 17 22:34:29.472267 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:29.472229    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" path="/var/lib/kubelet/pods/088a252d-4eef-421f-8dc9-c76232363bb8/volumes"
Sep 17 22:34:29.472814 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:29.472797    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" path="/var/lib/kubelet/pods/7a09fad3-de4d-4db3-8dd1-d5c98ff11725/volumes"
Sep 17 22:34:31.599244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:31.599203    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:32.025580 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025484    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:34:32.025795 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025780    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025796    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025807    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025813    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025820    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kube-rbac-proxy"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025826    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kube-rbac-proxy"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025831    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kube-rbac-proxy"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kube-rbac-proxy"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025848    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="storage-initializer"
Sep 17 22:34:32.025859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025853    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="storage-initializer"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025866    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="storage-initializer"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025871    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="storage-initializer"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025911    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kube-rbac-proxy"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025918    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kserve-container"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025923    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09fad3-de4d-4db3-8dd1-d5c98ff11725" containerName="kserve-container"
Sep 17 22:34:32.026292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.025928    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="088a252d-4eef-421f-8dc9-c76232363bb8" containerName="kube-rbac-proxy"
Sep 17 22:34:32.259857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.259815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:34:32.260039 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.259955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.266909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.266885    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 22:34:32.267051 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.267035    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 22:34:32.333570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.333515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.333747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.333598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.333747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.333661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bps2z\" (UniqueName: \"kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.333747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.333693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.333747 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.333719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435181 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bps2z\" (UniqueName: \"kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435181 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435438 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435438 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435438 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.435822 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.435793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.436041 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.436020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.436087 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.436020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.437814 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.437794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.444889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.444866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bps2z\" (UniqueName: \"kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z\") pod \"isvc-lightgbm-predictor-54f85ddf88-88t5p\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.569880 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.569839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:32.856606 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:32.856283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:34:32.859195 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:34:32.859166    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c93fb99_7bf6_4671_ac9e_c00830c34a33.slice/crio-f23162a7e6ca25d2a1c2be37c0037044558845e84fd6a4d49d21b33cde56d070 WatchSource:0}: Error finding container f23162a7e6ca25d2a1c2be37c0037044558845e84fd6a4d49d21b33cde56d070: Status 404 returned error can't find the container with id f23162a7e6ca25d2a1c2be37c0037044558845e84fd6a4d49d21b33cde56d070
Sep 17 22:34:33.714304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:33.714268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerStarted","Data":"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"}
Sep 17 22:34:33.714578 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:33.714325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerStarted","Data":"f23162a7e6ca25d2a1c2be37c0037044558845e84fd6a4d49d21b33cde56d070"}
Sep 17 22:34:37.729615 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:37.729578    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerID="d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f" exitCode=0
Sep 17 22:34:37.730022 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:37.729648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerDied","Data":"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"}
Sep 17 22:34:41.599272 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:41.599230    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:34:44.764302 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.764258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerStarted","Data":"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"}
Sep 17 22:34:44.764709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.764353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerStarted","Data":"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"}
Sep 17 22:34:44.764709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.764640    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:44.764709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.764685    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:44.765847 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.765820    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:34:44.786618 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:44.786569    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podStartSLOduration=2.504744378 podStartE2EDuration="12.786529502s" totalImagesPullingTime="6.397391834s" totalInitContainerRuntime="3.88439329s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:34:44.785759799 +0000 UTC m=+2189.453822570" watchObservedRunningTime="2026-09-17 22:34:44.786529502 +0000 UTC m=+2189.454592249"
Sep 17 22:34:45.768096 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:45.768056    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:34:50.772433 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:50.772402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:34:50.773061 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:50.773032    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:34:51.599165 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:34:51.599125    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:35:00.773476 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:00.773437    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:01.599282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:01.599241    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:35:10.773165 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:10.773119    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:11.599782 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:11.599744    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 22:35:19.472466 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:19.472437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:35:20.773437 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:20.773398    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:21.661916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:21.661883    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:35:21.662135 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:21.662113    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container" containerID="cri-o://4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5" gracePeriod=30
Sep 17 22:35:21.662225 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:21.662159    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kube-rbac-proxy" containerID="cri-o://a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483" gracePeriod=30
Sep 17 22:35:21.902539 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:21.902508    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerID="a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483" exitCode=2
Sep 17 22:35:21.902895 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:21.902590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerDied","Data":"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"}
Sep 17 22:35:26.102767 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.102737    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:35:26.153655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location\") pod \"ae45340f-584f-4985-9962-a4c8f694b5c5\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") "
Sep 17 22:35:26.153655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config\") pod \"ae45340f-584f-4985-9962-a4c8f694b5c5\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") "
Sep 17 22:35:26.153911 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153717    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qcz7\" (UniqueName: \"kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7\") pod \"ae45340f-584f-4985-9962-a4c8f694b5c5\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") "
Sep 17 22:35:26.153911 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls\") pod \"ae45340f-584f-4985-9962-a4c8f694b5c5\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") "
Sep 17 22:35:26.153911 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert\") pod \"ae45340f-584f-4985-9962-a4c8f694b5c5\" (UID: \"ae45340f-584f-4985-9962-a4c8f694b5c5\") "
Sep 17 22:35:26.154042 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.153979    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae45340f-584f-4985-9962-a4c8f694b5c5" (UID: "ae45340f-584f-4985-9962-a4c8f694b5c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:26.154138 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.154098    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae45340f-584f-4985-9962-a4c8f694b5c5" (UID: "ae45340f-584f-4985-9962-a4c8f694b5c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:35:26.154256 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.154181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae45340f-584f-4985-9962-a4c8f694b5c5" (UID: "ae45340f-584f-4985-9962-a4c8f694b5c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:35:26.155914 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.155894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7" (OuterVolumeSpecName: "kube-api-access-2qcz7") pod "ae45340f-584f-4985-9962-a4c8f694b5c5" (UID: "ae45340f-584f-4985-9962-a4c8f694b5c5"). InnerVolumeSpecName "kube-api-access-2qcz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:35:26.156011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.155973    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae45340f-584f-4985-9962-a4c8f694b5c5" (UID: "ae45340f-584f-4985-9962-a4c8f694b5c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:35:26.254381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.254291    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae45340f-584f-4985-9962-a4c8f694b5c5-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:35:26.254381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.254322    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:35:26.254381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.254332    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcz7\" (UniqueName: \"kubernetes.io/projected/ae45340f-584f-4985-9962-a4c8f694b5c5-kube-api-access-2qcz7\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:35:26.254381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.254343    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae45340f-584f-4985-9962-a4c8f694b5c5-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:35:26.254381 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.254352    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae45340f-584f-4985-9962-a4c8f694b5c5-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:35:26.922736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.922691    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerID="4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5" exitCode=0
Sep 17 22:35:26.922948 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.922740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerDied","Data":"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"}
Sep 17 22:35:26.922948 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.922809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9" event={"ID":"ae45340f-584f-4985-9962-a4c8f694b5c5","Type":"ContainerDied","Data":"71b2a650a7654cea55e4be071bfad67e920ec49a810d6d01d20735283c091448"}
Sep 17 22:35:26.922948 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.922755    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"
Sep 17 22:35:26.922948 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.922838    2582 scope.go:117] "RemoveContainer" containerID="a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"
Sep 17 22:35:26.931684 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.931661    2582 scope.go:117] "RemoveContainer" containerID="4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"
Sep 17 22:35:26.941001 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.940977    2582 scope.go:117] "RemoveContainer" containerID="2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"
Sep 17 22:35:26.944669 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.944641    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:35:26.948418 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.948389    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-v6lj9"]
Sep 17 22:35:26.949410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.949387    2582 scope.go:117] "RemoveContainer" containerID="a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"
Sep 17 22:35:26.949782 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:35:26.949763    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483\": container with ID starting with a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483 not found: ID does not exist" containerID="a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"
Sep 17 22:35:26.949838 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.949792    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483"} err="failed to get container status \"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483\": rpc error: code = NotFound desc = could not find container \"a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483\": container with ID starting with a7f8cc30eb2d5e8f078ebe354ac69e4bcfdef2edc67f8962dede2443b4b15483 not found: ID does not exist"
Sep 17 22:35:26.949838 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.949812    2582 scope.go:117] "RemoveContainer" containerID="4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"
Sep 17 22:35:26.950108 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:35:26.950088    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5\": container with ID starting with 4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5 not found: ID does not exist" containerID="4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"
Sep 17 22:35:26.950189 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.950118    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5"} err="failed to get container status \"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5\": rpc error: code = NotFound desc = could not find container \"4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5\": container with ID starting with 4040b22627492114191364227a5d2693d0f6469dc8322c698c90b7a89df43fd5 not found: ID does not exist"
Sep 17 22:35:26.950189 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.950140    2582 scope.go:117] "RemoveContainer" containerID="2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"
Sep 17 22:35:26.950402 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:35:26.950382    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833\": container with ID starting with 2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833 not found: ID does not exist" containerID="2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"
Sep 17 22:35:26.950440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:26.950416    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833"} err="failed to get container status \"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833\": rpc error: code = NotFound desc = could not find container \"2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833\": container with ID starting with 2dfb1ff52fb46d767cc8ffad5bcf9c3729abcb20310b67ef6ecd2c1787829833 not found: ID does not exist"
Sep 17 22:35:27.472262 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:27.472220    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" path="/var/lib/kubelet/pods/ae45340f-584f-4985-9962-a4c8f694b5c5/volumes"
Sep 17 22:35:30.773285 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:30.773203    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:40.773320 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:40.773278    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:50.773463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:50.773421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:35:54.468353 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:35:54.468301    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:36:04.468798 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:04.468754    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:36:14.468659 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:14.468604    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:36:24.469739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:24.469697    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:36:32.740047 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:32.740013    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:36:32.740527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:32.740248    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" containerID="cri-o://c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33" gracePeriod=30
Sep 17 22:36:32.740527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:32.740323    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kube-rbac-proxy" containerID="cri-o://042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb" gracePeriod=30
Sep 17 22:36:33.156570 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:33.156516    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerID="042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb" exitCode=2
Sep 17 22:36:33.156773 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:33.156585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerDied","Data":"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"}
Sep 17 22:36:34.468486 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:34.468438    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 22:36:35.768772 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:35.768726    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 22:36:37.936818 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.936782    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kube-rbac-proxy"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kube-rbac-proxy"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937058    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="storage-initializer"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="storage-initializer"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937072    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937078    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937135    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kserve-container"
Sep 17 22:36:37.937198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.937142    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae45340f-584f-4985-9962-a4c8f694b5c5" containerName="kube-rbac-proxy"
Sep 17 22:36:37.939901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.939881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:37.942758 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.942737    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:36:37.942870 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.942799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 22:36:37.950532 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.950506    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:36:37.997198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.997165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvxb\" (UniqueName: \"kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:37.997198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.997210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:37.997420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.997238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:37.997420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.997292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:37.997420 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:37.997328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098009 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.097966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098009 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvxb\" (UniqueName: \"kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098615 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098813 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.098861 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.098796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.100487 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.100471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.106523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.106501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvxb\" (UniqueName: \"kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.256229 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.256176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:38.531722 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.531686    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:36:38.534158 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:36:38.534132    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5b1888_0297_48bc_9621_f3059e25fff3.slice/crio-f9e8fad26d427666c5de43fc62cbb2beb9bfe8166066941dbb605e5adbbc4660 WatchSource:0}: Error finding container f9e8fad26d427666c5de43fc62cbb2beb9bfe8166066941dbb605e5adbbc4660: Status 404 returned error can't find the container with id f9e8fad26d427666c5de43fc62cbb2beb9bfe8166066941dbb605e5adbbc4660
Sep 17 22:36:38.571184 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.571158    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:36:38.602429 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602399    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config\") pod \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") "
Sep 17 22:36:38.602587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602467    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location\") pod \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") "
Sep 17 22:36:38.602587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bps2z\" (UniqueName: \"kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z\") pod \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") "
Sep 17 22:36:38.602587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602582    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert\") pod \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") "
Sep 17 22:36:38.603211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls\") pod \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\" (UID: \"2c93fb99-7bf6-4671-ac9e-c00830c34a33\") "
Sep 17 22:36:38.603211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c93fb99-7bf6-4671-ac9e-c00830c34a33" (UID: "2c93fb99-7bf6-4671-ac9e-c00830c34a33"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:36:38.603211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.602852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c93fb99-7bf6-4671-ac9e-c00830c34a33" (UID: "2c93fb99-7bf6-4671-ac9e-c00830c34a33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:36:38.603211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.603031    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c93fb99-7bf6-4671-ac9e-c00830c34a33" (UID: "2c93fb99-7bf6-4671-ac9e-c00830c34a33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:36:38.605800 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.605777    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c93fb99-7bf6-4671-ac9e-c00830c34a33" (UID: "2c93fb99-7bf6-4671-ac9e-c00830c34a33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:36:38.605919 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.605906    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z" (OuterVolumeSpecName: "kube-api-access-bps2z") pod "2c93fb99-7bf6-4671-ac9e-c00830c34a33" (UID: "2c93fb99-7bf6-4671-ac9e-c00830c34a33"). InnerVolumeSpecName "kube-api-access-bps2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:36:38.703584 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.703533    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:36:38.703584 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.703581    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c93fb99-7bf6-4671-ac9e-c00830c34a33-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:36:38.703792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.703595    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:36:38.703792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.703609    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:36:38.703792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:38.703622    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bps2z\" (UniqueName: \"kubernetes.io/projected/2c93fb99-7bf6-4671-ac9e-c00830c34a33-kube-api-access-bps2z\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:36:39.181031 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.180992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerStarted","Data":"04c1335da23ec3f1a92e4734ff1038daf873212c37f962236f1a00e24397d3ff"}
Sep 17 22:36:39.181485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.181066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerStarted","Data":"f9e8fad26d427666c5de43fc62cbb2beb9bfe8166066941dbb605e5adbbc4660"}
Sep 17 22:36:39.183097 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.183068    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerID="c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33" exitCode=0
Sep 17 22:36:39.183211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.183120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerDied","Data":"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"}
Sep 17 22:36:39.183211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.183132    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"
Sep 17 22:36:39.183211 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.183177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p" event={"ID":"2c93fb99-7bf6-4671-ac9e-c00830c34a33","Type":"ContainerDied","Data":"f23162a7e6ca25d2a1c2be37c0037044558845e84fd6a4d49d21b33cde56d070"}
Sep 17 22:36:39.183320 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.183215    2582 scope.go:117] "RemoveContainer" containerID="042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"
Sep 17 22:36:39.191893 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.191871    2582 scope.go:117] "RemoveContainer" containerID="c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"
Sep 17 22:36:39.199643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.199620    2582 scope.go:117] "RemoveContainer" containerID="d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"
Sep 17 22:36:39.207221 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.207201    2582 scope.go:117] "RemoveContainer" containerID="042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"
Sep 17 22:36:39.207599 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:36:39.207572    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb\": container with ID starting with 042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb not found: ID does not exist" containerID="042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"
Sep 17 22:36:39.207688 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.207605    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb"} err="failed to get container status \"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb\": rpc error: code = NotFound desc = could not find container \"042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb\": container with ID starting with 042d4accd1069663e9c3bac57ad188d98e6c793bebfaa5cde126b20aec299ecb not found: ID does not exist"
Sep 17 22:36:39.207688 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.207623    2582 scope.go:117] "RemoveContainer" containerID="c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"
Sep 17 22:36:39.207894 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:36:39.207876    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33\": container with ID starting with c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33 not found: ID does not exist" containerID="c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"
Sep 17 22:36:39.207973 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.207904    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33"} err="failed to get container status \"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33\": rpc error: code = NotFound desc = could not find container \"c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33\": container with ID starting with c086d94ed8b627ca96bf5d139ba84e504e1eb76c198f02106c8b679191dd9c33 not found: ID does not exist"
Sep 17 22:36:39.207973 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.207927    2582 scope.go:117] "RemoveContainer" containerID="d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"
Sep 17 22:36:39.208199 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:36:39.208179    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f\": container with ID starting with d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f not found: ID does not exist" containerID="d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"
Sep 17 22:36:39.208294 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.208202    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f"} err="failed to get container status \"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f\": rpc error: code = NotFound desc = could not find container \"d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f\": container with ID starting with d62f383cc13b38abf8fd88515683a6f1583e20917f63bc9040c05db11c11839f not found: ID does not exist"
Sep 17 22:36:39.212655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.212630    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:36:39.217729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.217703    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-88t5p"]
Sep 17 22:36:39.472820 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:39.472735    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" path="/var/lib/kubelet/pods/2c93fb99-7bf6-4671-ac9e-c00830c34a33/volumes"
Sep 17 22:36:43.199675 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:43.199643    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerID="04c1335da23ec3f1a92e4734ff1038daf873212c37f962236f1a00e24397d3ff" exitCode=0
Sep 17 22:36:43.200079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:43.199725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerDied","Data":"04c1335da23ec3f1a92e4734ff1038daf873212c37f962236f1a00e24397d3ff"}
Sep 17 22:36:44.205918 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:44.205879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerStarted","Data":"b57236aca78013b12006c0a30679e00ee608322c40e83c6410a69da23317422c"}
Sep 17 22:36:44.206403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:44.205952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerStarted","Data":"c735132e5a325bf8a40c537c8954fc1aa64c8401254555b2ebbbec2664bd384c"}
Sep 17 22:36:44.206403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:44.206188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:44.225265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:44.225211    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podStartSLOduration=3.442238552 podStartE2EDuration="7.225195367s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782956815s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:36:44.223652972 +0000 UTC m=+2308.891715733" watchObservedRunningTime="2026-09-17 22:36:44.225195367 +0000 UTC m=+2308.893258113"
Sep 17 22:36:45.210168 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:45.210141    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:45.211354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:45.211325    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:36:46.213164 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:46.213125    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:36:51.217587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:51.217536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:36:51.218081 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:36:51.218052    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:01.218535 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:01.218445    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:11.218019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:11.217980    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:17.370956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.370920    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371186    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371199    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371213    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kube-rbac-proxy"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371219    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kube-rbac-proxy"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="storage-initializer"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="storage-initializer"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371282    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kube-rbac-proxy"
Sep 17 22:37:17.371365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.371289    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c93fb99-7bf6-4671-ac9e-c00830c34a33" containerName="kserve-container"
Sep 17 22:37:17.374193 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.374175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.376920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.376889    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 22:37:17.377075 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.376891    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:37:17.383849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.383821    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:37:17.488828 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.488793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5vk\" (UniqueName: \"kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.489007 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.488843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.489007 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.488866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.489007 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.488920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.489007 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.488945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590259 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590259 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590535 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590535 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590535 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5vk\" (UniqueName: \"kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.590823 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.590801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.591028 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.591010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.591074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.591060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.592833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.592814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.599183 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.599160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5vk\" (UniqueName: \"kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lptf2\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.685193 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.685090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:17.955930 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:17.955906    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:37:17.958135 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:37:17.958105    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda529d665_c887_4397_b280_ff5705648392.slice/crio-0c0e7ff6de0b033b9e6bd41f5a21eafdcd2b74c27b7452ac2fc218b5e30826e7 WatchSource:0}: Error finding container 0c0e7ff6de0b033b9e6bd41f5a21eafdcd2b74c27b7452ac2fc218b5e30826e7: Status 404 returned error can't find the container with id 0c0e7ff6de0b033b9e6bd41f5a21eafdcd2b74c27b7452ac2fc218b5e30826e7
Sep 17 22:37:18.329733 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:18.329687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerStarted","Data":"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"}
Sep 17 22:37:18.329906 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:18.329793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerStarted","Data":"0c0e7ff6de0b033b9e6bd41f5a21eafdcd2b74c27b7452ac2fc218b5e30826e7"}
Sep 17 22:37:21.217988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:21.217948    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:22.346686 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:22.346651    2582 generic.go:356] "Generic (PLEG): container finished" podID="a529d665-c887-4397-b280-ff5705648392" containerID="77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7" exitCode=0
Sep 17 22:37:22.347034 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:22.346700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerDied","Data":"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"}
Sep 17 22:37:23.352977 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.352938    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerStarted","Data":"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"}
Sep 17 22:37:23.353385 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.353031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerStarted","Data":"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"}
Sep 17 22:37:23.353385 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.353370    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:23.353498 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.353432    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:23.354629 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.354605    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:37:23.372320 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:23.372253    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podStartSLOduration=2.5107180099999997 podStartE2EDuration="6.3722328s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86151479s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:37:23.37157808 +0000 UTC m=+2348.039640845" watchObservedRunningTime="2026-09-17 22:37:23.3722328 +0000 UTC m=+2348.040295548"
Sep 17 22:37:24.357434 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:24.357397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:37:29.363955 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:29.363926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:37:29.364433 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:29.364403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:37:31.218916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:31.218877    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:39.364684 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:39.364638    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:37:41.218738 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:41.218694    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:49.364407 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:49.364357    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:37:51.218214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:51.218167    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:37:59.364839 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:37:59.364794    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:38:01.218326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:01.218282    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:38:09.364941 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:09.364899    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:38:11.218115 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:11.218075    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:38:11.473245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:11.473143    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:38:19.364623 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:19.364572    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:38:21.473711 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:21.473680    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:38:29.364989 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:29.364896    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 22:38:29.640518 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:29.640427    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:38:29.640743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:29.640693    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" containerID="cri-o://c735132e5a325bf8a40c537c8954fc1aa64c8401254555b2ebbbec2664bd384c" gracePeriod=30
Sep 17 22:38:29.640962 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:29.640714    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kube-rbac-proxy" containerID="cri-o://b57236aca78013b12006c0a30679e00ee608322c40e83c6410a69da23317422c" gracePeriod=30
Sep 17 22:38:30.608629 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:30.608596    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerID="b57236aca78013b12006c0a30679e00ee608322c40e83c6410a69da23317422c" exitCode=2
Sep 17 22:38:30.609025 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:30.608648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerDied","Data":"b57236aca78013b12006c0a30679e00ee608322c40e83c6410a69da23317422c"}
Sep 17 22:38:31.214454 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:31.214412    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 22:38:31.473739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:31.473654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 22:38:34.469428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:34.469395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:38:35.627779 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:35.627740    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerID="c735132e5a325bf8a40c537c8954fc1aa64c8401254555b2ebbbec2664bd384c" exitCode=0
Sep 17 22:38:35.628172 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:35.627808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerDied","Data":"c735132e5a325bf8a40c537c8954fc1aa64c8401254555b2ebbbec2664bd384c"}
Sep 17 22:38:35.882277 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:35.882215    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:38:36.008087 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008044    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config\") pod \"0d5b1888-0297-48bc-9621-f3059e25fff3\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") "
Sep 17 22:38:36.008087 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008097    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert\") pod \"0d5b1888-0297-48bc-9621-f3059e25fff3\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") "
Sep 17 22:38:36.008358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location\") pod \"0d5b1888-0297-48bc-9621-f3059e25fff3\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") "
Sep 17 22:38:36.008358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008158    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bvxb\" (UniqueName: \"kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb\") pod \"0d5b1888-0297-48bc-9621-f3059e25fff3\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") "
Sep 17 22:38:36.008358 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls\") pod \"0d5b1888-0297-48bc-9621-f3059e25fff3\" (UID: \"0d5b1888-0297-48bc-9621-f3059e25fff3\") "
Sep 17 22:38:36.008541 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008511    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d5b1888-0297-48bc-9621-f3059e25fff3" (UID: "0d5b1888-0297-48bc-9621-f3059e25fff3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:38:36.008633 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d5b1888-0297-48bc-9621-f3059e25fff3" (UID: "0d5b1888-0297-48bc-9621-f3059e25fff3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:38:36.008633 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.008517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d5b1888-0297-48bc-9621-f3059e25fff3" (UID: "0d5b1888-0297-48bc-9621-f3059e25fff3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:36.010350 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.010318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d5b1888-0297-48bc-9621-f3059e25fff3" (UID: "0d5b1888-0297-48bc-9621-f3059e25fff3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:38:36.010458 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.010377    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb" (OuterVolumeSpecName: "kube-api-access-7bvxb") pod "0d5b1888-0297-48bc-9621-f3059e25fff3" (UID: "0d5b1888-0297-48bc-9621-f3059e25fff3"). InnerVolumeSpecName "kube-api-access-7bvxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:38:36.109561 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.109519    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d5b1888-0297-48bc-9621-f3059e25fff3-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:36.109737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.109572    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvxb\" (UniqueName: \"kubernetes.io/projected/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-api-access-7bvxb\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:36.109737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.109583    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d5b1888-0297-48bc-9621-f3059e25fff3-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:36.109737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.109593    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:36.109737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.109602    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d5b1888-0297-48bc-9621-f3059e25fff3-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:36.633409 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.633374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb" event={"ID":"0d5b1888-0297-48bc-9621-f3059e25fff3","Type":"ContainerDied","Data":"f9e8fad26d427666c5de43fc62cbb2beb9bfe8166066941dbb605e5adbbc4660"}
Sep 17 22:38:36.633889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.633423    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"
Sep 17 22:38:36.633889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.633441    2582 scope.go:117] "RemoveContainer" containerID="b57236aca78013b12006c0a30679e00ee608322c40e83c6410a69da23317422c"
Sep 17 22:38:36.642470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.642444    2582 scope.go:117] "RemoveContainer" containerID="c735132e5a325bf8a40c537c8954fc1aa64c8401254555b2ebbbec2664bd384c"
Sep 17 22:38:36.649784 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.649765    2582 scope.go:117] "RemoveContainer" containerID="04c1335da23ec3f1a92e4734ff1038daf873212c37f962236f1a00e24397d3ff"
Sep 17 22:38:36.655890 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.655860    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:38:36.659079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:36.659055    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f4pzb"]
Sep 17 22:38:37.472907 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:37.472871    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" path="/var/lib/kubelet/pods/0d5b1888-0297-48bc-9621-f3059e25fff3/volumes"
Sep 17 22:38:37.948836 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:37.948803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:38:37.949271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:37.949049    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container" containerID="cri-o://3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f" gracePeriod=30
Sep 17 22:38:37.949271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:37.949076    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kube-rbac-proxy" containerID="cri-o://ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f" gracePeriod=30
Sep 17 22:38:38.644923 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:38.644886    2582 generic.go:356] "Generic (PLEG): container finished" podID="a529d665-c887-4397-b280-ff5705648392" containerID="ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f" exitCode=2
Sep 17 22:38:38.645087 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:38.644959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerDied","Data":"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"}
Sep 17 22:38:39.358523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:39.358473    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 22:38:42.393724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.393696    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:38:42.460756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.460714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls\") pod \"a529d665-c887-4397-b280-ff5705648392\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") "
Sep 17 22:38:42.460756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.460759    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config\") pod \"a529d665-c887-4397-b280-ff5705648392\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") "
Sep 17 22:38:42.461024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.460785    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert\") pod \"a529d665-c887-4397-b280-ff5705648392\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") "
Sep 17 22:38:42.461024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.460812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location\") pod \"a529d665-c887-4397-b280-ff5705648392\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") "
Sep 17 22:38:42.461024 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.460867    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc5vk\" (UniqueName: \"kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk\") pod \"a529d665-c887-4397-b280-ff5705648392\" (UID: \"a529d665-c887-4397-b280-ff5705648392\") "
Sep 17 22:38:42.461194 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.461168    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a529d665-c887-4397-b280-ff5705648392" (UID: "a529d665-c887-4397-b280-ff5705648392"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:38:42.461243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.461175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a529d665-c887-4397-b280-ff5705648392" (UID: "a529d665-c887-4397-b280-ff5705648392"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:38:42.461243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.461176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a529d665-c887-4397-b280-ff5705648392" (UID: "a529d665-c887-4397-b280-ff5705648392"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:42.462956 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.462915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a529d665-c887-4397-b280-ff5705648392" (UID: "a529d665-c887-4397-b280-ff5705648392"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:38:42.463069 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.463040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk" (OuterVolumeSpecName: "kube-api-access-bc5vk") pod "a529d665-c887-4397-b280-ff5705648392" (UID: "a529d665-c887-4397-b280-ff5705648392"). InnerVolumeSpecName "kube-api-access-bc5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:38:42.561744 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.561710    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a529d665-c887-4397-b280-ff5705648392-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:42.561744 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.561744    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:42.561931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.561755    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a529d665-c887-4397-b280-ff5705648392-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:42.561931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.561766    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a529d665-c887-4397-b280-ff5705648392-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:42.561931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.561774    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5vk\" (UniqueName: \"kubernetes.io/projected/a529d665-c887-4397-b280-ff5705648392-kube-api-access-bc5vk\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:38:42.667941 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.667898    2582 generic.go:356] "Generic (PLEG): container finished" podID="a529d665-c887-4397-b280-ff5705648392" containerID="3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f" exitCode=0
Sep 17 22:38:42.668114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.667955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerDied","Data":"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"}
Sep 17 22:38:42.668114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.667971    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"
Sep 17 22:38:42.668114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.668046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2" event={"ID":"a529d665-c887-4397-b280-ff5705648392","Type":"ContainerDied","Data":"0c0e7ff6de0b033b9e6bd41f5a21eafdcd2b74c27b7452ac2fc218b5e30826e7"}
Sep 17 22:38:42.668114 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.668082    2582 scope.go:117] "RemoveContainer" containerID="ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"
Sep 17 22:38:42.676446 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.676426    2582 scope.go:117] "RemoveContainer" containerID="3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"
Sep 17 22:38:42.684017 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.683998    2582 scope.go:117] "RemoveContainer" containerID="77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"
Sep 17 22:38:42.689831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.689808    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:38:42.691719 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.691699    2582 scope.go:117] "RemoveContainer" containerID="ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"
Sep 17 22:38:42.692223 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:38:42.692192    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f\": container with ID starting with ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f not found: ID does not exist" containerID="ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"
Sep 17 22:38:42.692335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.692247    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f"} err="failed to get container status \"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f\": rpc error: code = NotFound desc = could not find container \"ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f\": container with ID starting with ec6b28b2e29c15fe21777c8b365ac73cea570ba4c3ddde7c0a895e9eb7bc4f7f not found: ID does not exist"
Sep 17 22:38:42.692335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.692275    2582 scope.go:117] "RemoveContainer" containerID="3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"
Sep 17 22:38:42.692625 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:38:42.692596    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f\": container with ID starting with 3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f not found: ID does not exist" containerID="3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"
Sep 17 22:38:42.692712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.692623    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f"} err="failed to get container status \"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f\": rpc error: code = NotFound desc = could not find container \"3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f\": container with ID starting with 3349c131638b1995de62b64841cdc494b1c7159ed70d014cdb4bd9a812d1198f not found: ID does not exist"
Sep 17 22:38:42.692712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.692644    2582 scope.go:117] "RemoveContainer" containerID="77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"
Sep 17 22:38:42.692938 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:38:42.692919    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7\": container with ID starting with 77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7 not found: ID does not exist" containerID="77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"
Sep 17 22:38:42.692992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.692946    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7"} err="failed to get container status \"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7\": rpc error: code = NotFound desc = could not find container \"77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7\": container with ID starting with 77eee01c4226cbd185ec9d9a2778b5d8fd0cd541547e9fea0a963a44defe7cb7 not found: ID does not exist"
Sep 17 22:38:42.693783 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:42.693764    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lptf2"]
Sep 17 22:38:43.472137 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:38:43.472099    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a529d665-c887-4397-b280-ff5705648392" path="/var/lib/kubelet/pods/a529d665-c887-4397-b280-ff5705648392/volumes"
Sep 17 22:39:30.103112 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103318    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kube-rbac-proxy"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103329    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kube-rbac-proxy"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103339    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kube-rbac-proxy"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103344    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kube-rbac-proxy"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103357    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103363    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103370    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103375    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103381    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="storage-initializer"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103386    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="storage-initializer"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103397    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="storage-initializer"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103402    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="storage-initializer"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103438    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103445    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kserve-container"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103452    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5b1888-0297-48bc-9621-f3059e25fff3" containerName="kube-rbac-proxy"
Sep 17 22:39:30.103628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.103458    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a529d665-c887-4397-b280-ff5705648392" containerName="kube-rbac-proxy"
Sep 17 22:39:30.106354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.106336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.109341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.109317    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:39:30.109491 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.109324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:39:30.109593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.109572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:39:30.110567 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.110535    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:39:30.110731 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.110708    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 22:39:30.110844 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.110609    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:39:30.110844 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.110666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 22:39:30.116032 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.115996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:39:30.206681 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.206651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc9q\" (UniqueName: \"kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.206681 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.206684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.206905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.206708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.206905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.206780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.206905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.206812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307734 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.307700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307734 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.307734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307966 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.307786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc9q\" (UniqueName: \"kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307966 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.307802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307966 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.307826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.307966 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:30.307936    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 17 22:39:30.308176 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:30.308036    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls podName:c452aa6d-5c7c-476e-b6fe-182032e93190 nodeName:}" failed. No retries permitted until 2026-09-17 22:39:30.808014926 +0000 UTC m=+2475.476077660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 17 22:39:30.308353 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.308321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.308461 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.308442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.308586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.308567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.316402 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.316380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc9q\" (UniqueName: \"kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.811851 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.811801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:30.814265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:30.814238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:31.017435 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:31.017378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:31.288598 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:31.288565    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:39:31.291906 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:39:31.291880    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc452aa6d_5c7c_476e_b6fe_182032e93190.slice/crio-13d366c51c868dbf5eb47b0ea8ee077b2e38f04562457f89a76d5532d1d92a86 WatchSource:0}: Error finding container 13d366c51c868dbf5eb47b0ea8ee077b2e38f04562457f89a76d5532d1d92a86: Status 404 returned error can't find the container with id 13d366c51c868dbf5eb47b0ea8ee077b2e38f04562457f89a76d5532d1d92a86
Sep 17 22:39:31.832758 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:31.832718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerStarted","Data":"da8759a49a1837f23cccfc8177e991d573beedde8d99d6261f8b41883fab76e4"}
Sep 17 22:39:31.832938 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:31.832791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerStarted","Data":"13d366c51c868dbf5eb47b0ea8ee077b2e38f04562457f89a76d5532d1d92a86"}
Sep 17 22:39:35.847556 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:35.847506    2582 generic.go:356] "Generic (PLEG): container finished" podID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerID="da8759a49a1837f23cccfc8177e991d573beedde8d99d6261f8b41883fab76e4" exitCode=0
Sep 17 22:39:35.847943 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:35.847577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerDied","Data":"da8759a49a1837f23cccfc8177e991d573beedde8d99d6261f8b41883fab76e4"}
Sep 17 22:39:36.853614 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:36.853566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerStarted","Data":"6faa5848d4f9029d15d3213f4e9e02b0534f94fcb52c83fc57289cdaf3de9f24"}
Sep 17 22:39:36.854011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:36.853688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerStarted","Data":"71096b269cae85d617d73998d1d35dcc9f52cec084af0a841bf1bc0ce8ca4ad9"}
Sep 17 22:39:36.854011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:36.853970    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:36.872438 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:36.872382    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podStartSLOduration=3.120000742 podStartE2EDuration="6.872365051s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752364309s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:39:36.871879054 +0000 UTC m=+2481.539941823" watchObservedRunningTime="2026-09-17 22:39:36.872365051 +0000 UTC m=+2481.540427797"
Sep 17 22:39:37.858012 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:37.857975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:37.859070 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:37.859043    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 22:39:38.861271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:38.861228    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 22:39:43.866237 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:43.866203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:43.866839 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:43.866813    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 22:39:48.364475 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.364438    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:39:48.367383 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.367366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.370152 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.370130    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 22:39:48.370152 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.370140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:39:48.380102 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.380078    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:39:48.439480 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.439447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.439677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.439513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.439677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.439539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.439677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.439591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5ln\" (UniqueName: \"kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.439677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.439646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540620 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.540583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.540640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.540672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5ln\" (UniqueName: \"kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.540702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.540737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.540973 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:48.540845    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:39:48.540973 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:48.540949    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls podName:34e6be6b-aeaf-46ba-9798-5a7de27a52c7 nodeName:}" failed. No retries permitted until 2026-09-17 22:39:49.04092504 +0000 UTC m=+2493.708987780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-h2cnw" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:39:48.541129 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.541108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.541365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.541348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.541428 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.541412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:48.549421 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:48.549396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5ln\" (UniqueName: \"kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:49.045321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:49.045282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:49.045518 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:49.045441    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:39:49.045518 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:39:49.045507    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls podName:34e6be6b-aeaf-46ba-9798-5a7de27a52c7 nodeName:}" failed. No retries permitted until 2026-09-17 22:39:50.045492529 +0000 UTC m=+2494.713555255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-h2cnw" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:39:50.052913 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.052860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:50.055300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.055277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h2cnw\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:50.177299 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.177257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:50.447853 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.447818    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:39:50.451734 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:39:50.451711    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e6be6b_aeaf_46ba_9798_5a7de27a52c7.slice/crio-52e1f1cb538549433efe452d3f6fe534d040bbb0b4149aee1cc62093c5aa94b2 WatchSource:0}: Error finding container 52e1f1cb538549433efe452d3f6fe534d040bbb0b4149aee1cc62093c5aa94b2: Status 404 returned error can't find the container with id 52e1f1cb538549433efe452d3f6fe534d040bbb0b4149aee1cc62093c5aa94b2
Sep 17 22:39:50.904141 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.904108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerStarted","Data":"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"}
Sep 17 22:39:50.904318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:50.904168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerStarted","Data":"52e1f1cb538549433efe452d3f6fe534d040bbb0b4149aee1cc62093c5aa94b2"}
Sep 17 22:39:53.867589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:53.867531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:39:54.920715 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:54.920679    2582 generic.go:356] "Generic (PLEG): container finished" podID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerID="25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76" exitCode=0
Sep 17 22:39:54.921122 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:54.920750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerDied","Data":"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"}
Sep 17 22:39:55.926691 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:55.926658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerStarted","Data":"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"}
Sep 17 22:39:55.927068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:55.926716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerStarted","Data":"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"}
Sep 17 22:39:55.927068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:55.926901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:55.948134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:55.948086    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podStartSLOduration=4.284514785 podStartE2EDuration="7.948070267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663555482s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:39:55.946651457 +0000 UTC m=+2500.614714216" watchObservedRunningTime="2026-09-17 22:39:55.948070267 +0000 UTC m=+2500.616133013"
Sep 17 22:39:56.930772 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:56.930733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:39:56.931833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:56.931803    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:39:57.934291 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:39:57.934248    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:00.221379 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.221289    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:40:00.221842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.221581    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container" containerID="cri-o://71096b269cae85d617d73998d1d35dcc9f52cec084af0a841bf1bc0ce8ca4ad9" gracePeriod=30
Sep 17 22:40:00.221842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.221624    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kube-rbac-proxy" containerID="cri-o://6faa5848d4f9029d15d3213f4e9e02b0534f94fcb52c83fc57289cdaf3de9f24" gracePeriod=30
Sep 17 22:40:00.948136 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.948102    2582 generic.go:356] "Generic (PLEG): container finished" podID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerID="6faa5848d4f9029d15d3213f4e9e02b0534f94fcb52c83fc57289cdaf3de9f24" exitCode=2
Sep 17 22:40:00.948136 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.948132    2582 generic.go:356] "Generic (PLEG): container finished" podID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerID="71096b269cae85d617d73998d1d35dcc9f52cec084af0a841bf1bc0ce8ca4ad9" exitCode=0
Sep 17 22:40:00.948330 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.948172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerDied","Data":"6faa5848d4f9029d15d3213f4e9e02b0534f94fcb52c83fc57289cdaf3de9f24"}
Sep 17 22:40:00.948330 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.948255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerDied","Data":"71096b269cae85d617d73998d1d35dcc9f52cec084af0a841bf1bc0ce8ca4ad9"}
Sep 17 22:40:00.966468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:00.966440    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:40:01.043320 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043283    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert\") pod \"c452aa6d-5c7c-476e-b6fe-182032e93190\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") "
Sep 17 22:40:01.043320 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config\") pod \"c452aa6d-5c7c-476e-b6fe-182032e93190\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") "
Sep 17 22:40:01.043651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") pod \"c452aa6d-5c7c-476e-b6fe-182032e93190\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") "
Sep 17 22:40:01.043651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location\") pod \"c452aa6d-5c7c-476e-b6fe-182032e93190\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") "
Sep 17 22:40:01.043651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncc9q\" (UniqueName: \"kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q\") pod \"c452aa6d-5c7c-476e-b6fe-182032e93190\" (UID: \"c452aa6d-5c7c-476e-b6fe-182032e93190\") "
Sep 17 22:40:01.043831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c452aa6d-5c7c-476e-b6fe-182032e93190" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:40:01.043831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c452aa6d-5c7c-476e-b6fe-182032e93190" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:40:01.043831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.043797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c452aa6d-5c7c-476e-b6fe-182032e93190" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:40:01.045508 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.045481    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q" (OuterVolumeSpecName: "kube-api-access-ncc9q") pod "c452aa6d-5c7c-476e-b6fe-182032e93190" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190"). InnerVolumeSpecName "kube-api-access-ncc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:40:01.045644 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.045518    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c452aa6d-5c7c-476e-b6fe-182032e93190" (UID: "c452aa6d-5c7c-476e-b6fe-182032e93190"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:40:01.144665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.144628    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:40:01.144665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.144657    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:40:01.144665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.144667    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c452aa6d-5c7c-476e-b6fe-182032e93190-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:40:01.144665 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.144676    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c452aa6d-5c7c-476e-b6fe-182032e93190-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:40:01.145012 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.144685    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncc9q\" (UniqueName: \"kubernetes.io/projected/c452aa6d-5c7c-476e-b6fe-182032e93190-kube-api-access-ncc9q\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:40:01.953476 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.953440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt" event={"ID":"c452aa6d-5c7c-476e-b6fe-182032e93190","Type":"ContainerDied","Data":"13d366c51c868dbf5eb47b0ea8ee077b2e38f04562457f89a76d5532d1d92a86"}
Sep 17 22:40:01.953476 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.953458    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"
Sep 17 22:40:01.954047 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.953512    2582 scope.go:117] "RemoveContainer" containerID="6faa5848d4f9029d15d3213f4e9e02b0534f94fcb52c83fc57289cdaf3de9f24"
Sep 17 22:40:01.961589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.961568    2582 scope.go:117] "RemoveContainer" containerID="71096b269cae85d617d73998d1d35dcc9f52cec084af0a841bf1bc0ce8ca4ad9"
Sep 17 22:40:01.969187 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.969170    2582 scope.go:117] "RemoveContainer" containerID="da8759a49a1837f23cccfc8177e991d573beedde8d99d6261f8b41883fab76e4"
Sep 17 22:40:01.972386 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.972358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:40:01.977094 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:01.977059    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-r8cnt"]
Sep 17 22:40:02.939194 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:02.939165    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:40:02.939742 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:02.939702    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:03.472417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:03.472371    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" path="/var/lib/kubelet/pods/c452aa6d-5c7c-476e-b6fe-182032e93190/volumes"
Sep 17 22:40:12.939984 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:12.939940    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:22.940419 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:22.940377    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:32.940227 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:32.940182    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:42.939840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:42.939801    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:40:52.940112 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:40:52.940065    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:41:00.686846 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.686806    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687183    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container"
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687199    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container"
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687210    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kube-rbac-proxy"
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687219    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kube-rbac-proxy"
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687228    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="storage-initializer"
Sep 17 22:41:00.687280 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="storage-initializer"
Sep 17 22:41:00.687542 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687293    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kube-rbac-proxy"
Sep 17 22:41:00.687542 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.687304    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c452aa6d-5c7c-476e-b6fe-182032e93190" containerName="kserve-container"
Sep 17 22:41:00.690658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.690642    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.693198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.693170    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 22:41:00.693198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.693173    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 22:41:00.700734 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.700711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:41:00.777496 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.777458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrf8h\" (UniqueName: \"kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.777698 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.777509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.777698 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.777621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.777698 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.777651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.777698 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.777671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.878662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.878630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.878842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.878674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.878842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.878694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.878842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.878715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.878842 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.878758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf8h\" (UniqueName: \"kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.879169 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.879138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.879398 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.879379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.879485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.879413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.881242 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.881223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:00.887040 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:00.887018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf8h\" (UniqueName: \"kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h\") pod \"isvc-sklearn-mcp-predictor-67cd675847-n4trc\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:01.001408 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:01.001303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:01.276158 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:01.276033    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:41:01.281375 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:41:01.279262    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb94b533_d114_446b_a7c8_4fa3cadd04b9.slice/crio-57f09ed20463dbe5bef8afea6be332fbb64d5594e3c44fe5214ebf7636c4486c WatchSource:0}: Error finding container 57f09ed20463dbe5bef8afea6be332fbb64d5594e3c44fe5214ebf7636c4486c: Status 404 returned error can't find the container with id 57f09ed20463dbe5bef8afea6be332fbb64d5594e3c44fe5214ebf7636c4486c
Sep 17 22:41:02.177313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:02.177275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerStarted","Data":"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"}
Sep 17 22:41:02.177745 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:02.177338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerStarted","Data":"57f09ed20463dbe5bef8afea6be332fbb64d5594e3c44fe5214ebf7636c4486c"}
Sep 17 22:41:02.940658 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:02.940606    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:41:06.192421 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:06.192386    2582 generic.go:356] "Generic (PLEG): container finished" podID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerID="b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439" exitCode=0
Sep 17 22:41:06.192864 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:06.192462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerDied","Data":"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"}
Sep 17 22:41:06.270965 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:06.270946    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:41:07.199540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:07.199502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerStarted","Data":"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"}
Sep 17 22:41:09.209662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.209619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerStarted","Data":"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"}
Sep 17 22:41:09.210057 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.209720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerStarted","Data":"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"}
Sep 17 22:41:09.210057 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.209764    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:09.210057 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.209809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:09.210057 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.209834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:09.231495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:09.231392    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podStartSLOduration=2.643080801 podStartE2EDuration="9.231378178s" totalImagesPullingTime="2.694189166s" totalInitContainerRuntime="3.894108211s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:41:09.229910077 +0000 UTC m=+2573.897972845" watchObservedRunningTime="2026-09-17 22:41:09.231378178 +0000 UTC m=+2573.899440926"
Sep 17 22:41:12.940458 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:12.940426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:41:15.219543 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:15.219514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:41:19.139569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:19.137831    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:41:19.139569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:19.138314    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" containerID="cri-o://3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0" gracePeriod=30
Sep 17 22:41:19.139569 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:19.139323    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kube-rbac-proxy" containerID="cri-o://b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768" gracePeriod=30
Sep 17 22:41:20.252936 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:20.252902    2582 generic.go:356] "Generic (PLEG): container finished" podID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerID="b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768" exitCode=2
Sep 17 22:41:20.253308 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:20.252973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerDied","Data":"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"}
Sep 17 22:41:22.934850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:22.934800    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 22:41:22.940100 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:22.940061    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 22:41:23.571616 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.571590    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:41:23.660066 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660028    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location\") pod \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") "
Sep 17 22:41:23.660066 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert\") pod \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") "
Sep 17 22:41:23.660303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config\") pod \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") "
Sep 17 22:41:23.660303 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660176    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q5ln\" (UniqueName: \"kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln\") pod \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") "
Sep 17 22:41:23.660410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") pod \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\" (UID: \"34e6be6b-aeaf-46ba-9798-5a7de27a52c7\") "
Sep 17 22:41:23.660458 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34e6be6b-aeaf-46ba-9798-5a7de27a52c7" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:23.660458 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34e6be6b-aeaf-46ba-9798-5a7de27a52c7" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:41:23.660538 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660519    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34e6be6b-aeaf-46ba-9798-5a7de27a52c7" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:41:23.660674 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660657    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:41:23.660743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660679    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:41:23.660743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.660693    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:41:23.662363 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.662342    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln" (OuterVolumeSpecName: "kube-api-access-6q5ln") pod "34e6be6b-aeaf-46ba-9798-5a7de27a52c7" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7"). InnerVolumeSpecName "kube-api-access-6q5ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:41:23.662478 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.662414    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34e6be6b-aeaf-46ba-9798-5a7de27a52c7" (UID: "34e6be6b-aeaf-46ba-9798-5a7de27a52c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:41:23.761397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.761314    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q5ln\" (UniqueName: \"kubernetes.io/projected/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-kube-api-access-6q5ln\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:41:23.761397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:23.761356    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34e6be6b-aeaf-46ba-9798-5a7de27a52c7-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:41:24.269072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.269039    2582 generic.go:356] "Generic (PLEG): container finished" podID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerID="3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0" exitCode=0
Sep 17 22:41:24.269525 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.269097    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"
Sep 17 22:41:24.269525 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.269113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerDied","Data":"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"}
Sep 17 22:41:24.269525 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.269174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw" event={"ID":"34e6be6b-aeaf-46ba-9798-5a7de27a52c7","Type":"ContainerDied","Data":"52e1f1cb538549433efe452d3f6fe534d040bbb0b4149aee1cc62093c5aa94b2"}
Sep 17 22:41:24.269525 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.269200    2582 scope.go:117] "RemoveContainer" containerID="b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"
Sep 17 22:41:24.277905 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.277883    2582 scope.go:117] "RemoveContainer" containerID="3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"
Sep 17 22:41:24.285350 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.285332    2582 scope.go:117] "RemoveContainer" containerID="25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"
Sep 17 22:41:24.291599 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.291571    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:41:24.293109 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.293091    2582 scope.go:117] "RemoveContainer" containerID="b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"
Sep 17 22:41:24.293427 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:41:24.293408    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768\": container with ID starting with b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768 not found: ID does not exist" containerID="b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"
Sep 17 22:41:24.293472 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.293436    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768"} err="failed to get container status \"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768\": rpc error: code = NotFound desc = could not find container \"b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768\": container with ID starting with b43e905af34d1675afb8eb9a85ec2d2476f81b2dbd2f20cd65209c0518d19768 not found: ID does not exist"
Sep 17 22:41:24.293472 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.293456    2582 scope.go:117] "RemoveContainer" containerID="3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"
Sep 17 22:41:24.293797 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:41:24.293759    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0\": container with ID starting with 3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0 not found: ID does not exist" containerID="3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"
Sep 17 22:41:24.293849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.293803    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0"} err="failed to get container status \"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0\": rpc error: code = NotFound desc = could not find container \"3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0\": container with ID starting with 3d8a790e5fc38ecccb882ee679454d30b090b0bf3589f80a5c8be3092b0db0c0 not found: ID does not exist"
Sep 17 22:41:24.293849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.293819    2582 scope.go:117] "RemoveContainer" containerID="25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"
Sep 17 22:41:24.294078 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:41:24.294057    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76\": container with ID starting with 25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76 not found: ID does not exist" containerID="25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"
Sep 17 22:41:24.294121 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.294084    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76"} err="failed to get container status \"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76\": rpc error: code = NotFound desc = could not find container \"25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76\": container with ID starting with 25074838942af5f8228c711ceb497ad782ced5a6b562786b76f10496f8f26e76 not found: ID does not exist"
Sep 17 22:41:24.298171 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:24.298149    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h2cnw"]
Sep 17 22:41:25.472522 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:25.472480    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" path="/var/lib/kubelet/pods/34e6be6b-aeaf-46ba-9798-5a7de27a52c7/volumes"
Sep 17 22:41:41.002712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:41.002622    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 22:41:45.223617 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:41:45.223580    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:15.225021 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:15.224991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:20.749859 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:20.749826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:42:20.750297 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:20.750066    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container" containerID="cri-o://ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77" gracePeriod=30
Sep 17 22:42:20.750297 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:20.750109    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-agent" containerID="cri-o://6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581" gracePeriod=30
Sep 17 22:42:20.750297 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:20.750120    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" containerID="cri-o://854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f" gracePeriod=30
Sep 17 22:42:21.471737 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:21.471702    2582 generic.go:356] "Generic (PLEG): container finished" podID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerID="854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f" exitCode=2
Sep 17 22:42:21.472727 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:21.472704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerDied","Data":"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"}
Sep 17 22:42:23.481143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:23.481054    2582 generic.go:356] "Generic (PLEG): container finished" podID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerID="ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77" exitCode=0
Sep 17 22:42:23.481143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:23.481120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerDied","Data":"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"}
Sep 17 22:42:25.214848 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.214807    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:25.220401 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.220365    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 22:42:25.936251 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936213    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:42:25.936491 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936480    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="storage-initializer"
Sep 17 22:42:25.936537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936493    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="storage-initializer"
Sep 17 22:42:25.936537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936503    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kube-rbac-proxy"
Sep 17 22:42:25.936537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936508    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kube-rbac-proxy"
Sep 17 22:42:25.936537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936518    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container"
Sep 17 22:42:25.936537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936524    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container"
Sep 17 22:42:25.936728 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936614    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kube-rbac-proxy"
Sep 17 22:42:25.936728 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.936624    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e6be6b-aeaf-46ba-9798-5a7de27a52c7" containerName="kserve-container"
Sep 17 22:42:25.939637 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.939621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:25.942365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.942326    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 22:42:25.942490 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.942429    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 22:42:25.951681 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:25.951653    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:42:26.025151 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.025108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.025328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.025172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rns\" (UniqueName: \"kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.025328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.025207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.025328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.025243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.025328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.025279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.125888 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.125849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rns\" (UniqueName: \"kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.125904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.125943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.125979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126074 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.126027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126074 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:26.126058    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 22:42:26.126344 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:26.126144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls podName:c20959a8-c144-4679-93fd-bb1ea0904e88 nodeName:}" failed. No retries permitted until 2026-09-17 22:42:26.626117439 +0000 UTC m=+2651.294180174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-n5pk9" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 22:42:26.126484 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.126462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.126693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.126780 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.126761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.135262 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.135239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rns\" (UniqueName: \"kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.628999 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.628960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.631412 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.631388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-n5pk9\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:26.849643 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:26.849599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:27.122009 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:27.121977    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:42:27.125103 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:42:27.125068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20959a8_c144_4679_93fd_bb1ea0904e88.slice/crio-cb36c52a866f47776b2038fba57daec2ad3282e779db6f909df535340f01a8c9 WatchSource:0}: Error finding container cb36c52a866f47776b2038fba57daec2ad3282e779db6f909df535340f01a8c9: Status 404 returned error can't find the container with id cb36c52a866f47776b2038fba57daec2ad3282e779db6f909df535340f01a8c9
Sep 17 22:42:27.497292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:27.497198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerStarted","Data":"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"}
Sep 17 22:42:27.497292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:27.497280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerStarted","Data":"cb36c52a866f47776b2038fba57daec2ad3282e779db6f909df535340f01a8c9"}
Sep 17 22:42:30.214294 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:30.214245    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:31.512045 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:31.512005    2582 generic.go:356] "Generic (PLEG): container finished" podID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerID="ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a" exitCode=0
Sep 17 22:42:31.512427 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:31.512078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerDied","Data":"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"}
Sep 17 22:42:35.214599 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:35.214534    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:35.215011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:35.214733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:35.220632 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:35.220600    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 22:42:40.214443 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:40.214386    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:43.563491 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.563458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerStarted","Data":"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"}
Sep 17 22:42:43.563967 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.563519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerStarted","Data":"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"}
Sep 17 22:42:43.563967 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.563821    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:43.564088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.563976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:43.565447 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.565421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:42:43.585093 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:43.585024    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podStartSLOduration=2.67448742 podStartE2EDuration="18.584999859s" totalImagesPullingTime="11.661891094s" totalInitContainerRuntime="4.248621345s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:42:43.581735074 +0000 UTC m=+2668.249797839" watchObservedRunningTime="2026-09-17 22:42:43.584999859 +0000 UTC m=+2668.253062607"
Sep 17 22:42:44.567661 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:44.567616    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:42:45.214252 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:45.214203    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:45.220316 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:45.220274    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 22:42:45.220499 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:45.220441    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:49.571701 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:49.571673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:42:49.572173 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:49.572149    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:42:50.214855 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:50.214809    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 22:42:50.930843 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:50.930817    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:51.012695 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert\") pod \"db94b533-d114-446b-a7c8-4fa3cadd04b9\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") "
Sep 17 22:42:51.012695 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012659    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config\") pod \"db94b533-d114-446b-a7c8-4fa3cadd04b9\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") "
Sep 17 22:42:51.012886 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls\") pod \"db94b533-d114-446b-a7c8-4fa3cadd04b9\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") "
Sep 17 22:42:51.012928 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012885    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrf8h\" (UniqueName: \"kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h\") pod \"db94b533-d114-446b-a7c8-4fa3cadd04b9\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") "
Sep 17 22:42:51.012963 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location\") pod \"db94b533-d114-446b-a7c8-4fa3cadd04b9\" (UID: \"db94b533-d114-446b-a7c8-4fa3cadd04b9\") "
Sep 17 22:42:51.013010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012970    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db94b533-d114-446b-a7c8-4fa3cadd04b9" (UID: "db94b533-d114-446b-a7c8-4fa3cadd04b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:42:51.013010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.012969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db94b533-d114-446b-a7c8-4fa3cadd04b9" (UID: "db94b533-d114-446b-a7c8-4fa3cadd04b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:42:51.013160 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.013142    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:42:51.013200 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.013166    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db94b533-d114-446b-a7c8-4fa3cadd04b9-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:42:51.013357 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.013331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db94b533-d114-446b-a7c8-4fa3cadd04b9" (UID: "db94b533-d114-446b-a7c8-4fa3cadd04b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:42:51.014972 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.014948    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db94b533-d114-446b-a7c8-4fa3cadd04b9" (UID: "db94b533-d114-446b-a7c8-4fa3cadd04b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:42:51.015085 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.014995    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h" (OuterVolumeSpecName: "kube-api-access-mrf8h") pod "db94b533-d114-446b-a7c8-4fa3cadd04b9" (UID: "db94b533-d114-446b-a7c8-4fa3cadd04b9"). InnerVolumeSpecName "kube-api-access-mrf8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:42:51.113505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.113469    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db94b533-d114-446b-a7c8-4fa3cadd04b9-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:42:51.113505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.113501    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db94b533-d114-446b-a7c8-4fa3cadd04b9-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:42:51.113505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.113511    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrf8h\" (UniqueName: \"kubernetes.io/projected/db94b533-d114-446b-a7c8-4fa3cadd04b9-kube-api-access-mrf8h\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:42:51.594670 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.594636    2582 generic.go:356] "Generic (PLEG): container finished" podID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerID="6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581" exitCode=137
Sep 17 22:42:51.594841 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.594682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerDied","Data":"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"}
Sep 17 22:42:51.594841 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.594712    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"
Sep 17 22:42:51.594841 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.594738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc" event={"ID":"db94b533-d114-446b-a7c8-4fa3cadd04b9","Type":"ContainerDied","Data":"57f09ed20463dbe5bef8afea6be332fbb64d5594e3c44fe5214ebf7636c4486c"}
Sep 17 22:42:51.594841 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.594776    2582 scope.go:117] "RemoveContainer" containerID="854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"
Sep 17 22:42:51.602987 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.602964    2582 scope.go:117] "RemoveContainer" containerID="6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"
Sep 17 22:42:51.610342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.610325    2582 scope.go:117] "RemoveContainer" containerID="ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"
Sep 17 22:42:51.614787 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.614763    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:42:51.618382 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.618357    2582 scope.go:117] "RemoveContainer" containerID="b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"
Sep 17 22:42:51.618896 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.618872    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-n4trc"]
Sep 17 22:42:51.627598 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.627577    2582 scope.go:117] "RemoveContainer" containerID="854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"
Sep 17 22:42:51.628164 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:51.628142    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f\": container with ID starting with 854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f not found: ID does not exist" containerID="854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"
Sep 17 22:42:51.628279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628173    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f"} err="failed to get container status \"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f\": rpc error: code = NotFound desc = could not find container \"854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f\": container with ID starting with 854e6865150dc5d0e980cb340b3f467531fc68726128468f6a101ac802d4084f not found: ID does not exist"
Sep 17 22:42:51.628279 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628194    2582 scope.go:117] "RemoveContainer" containerID="6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"
Sep 17 22:42:51.628479 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:51.628462    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581\": container with ID starting with 6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581 not found: ID does not exist" containerID="6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"
Sep 17 22:42:51.628521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628484    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581"} err="failed to get container status \"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581\": rpc error: code = NotFound desc = could not find container \"6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581\": container with ID starting with 6f679f832822456fa688ebc4ab987fb8e8d5dc2645afcdb89b9dd0aa714cd581 not found: ID does not exist"
Sep 17 22:42:51.628521 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628503    2582 scope.go:117] "RemoveContainer" containerID="ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"
Sep 17 22:42:51.628787 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:51.628771    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77\": container with ID starting with ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77 not found: ID does not exist" containerID="ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"
Sep 17 22:42:51.628827 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628792    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77"} err="failed to get container status \"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77\": rpc error: code = NotFound desc = could not find container \"ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77\": container with ID starting with ccd48f3b289d55040824e1592641ca1a6be3a8cdba44c04d15b207603e10fd77 not found: ID does not exist"
Sep 17 22:42:51.628827 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.628809    2582 scope.go:117] "RemoveContainer" containerID="b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"
Sep 17 22:42:51.629035 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:42:51.629009    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439\": container with ID starting with b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439 not found: ID does not exist" containerID="b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"
Sep 17 22:42:51.629090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:51.629032    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439"} err="failed to get container status \"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439\": rpc error: code = NotFound desc = could not find container \"b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439\": container with ID starting with b016812d2257b65e296e9311a7f8f4db893c5494eff621214b71705747187439 not found: ID does not exist"
Sep 17 22:42:53.471889 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:53.471856    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" path="/var/lib/kubelet/pods/db94b533-d114-446b-a7c8-4fa3cadd04b9/volumes"
Sep 17 22:42:54.640187 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640146    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640402    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640412    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640425    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="storage-initializer"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640431    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="storage-initializer"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640440    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640446    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-agent"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640458    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-agent"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640502    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kube-rbac-proxy"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640510    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-container"
Sep 17 22:42:54.640682 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.640515    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db94b533-d114-446b-a7c8-4fa3cadd04b9" containerName="kserve-agent"
Sep 17 22:42:54.645059 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.645039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.647785 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.647759    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 22:42:54.647785 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.647776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 22:42:54.647976 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.647773    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 22:42:54.654709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.654683    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:42:54.743327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.743287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.743327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.743339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p6f\" (UniqueName: \"kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.743616 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.743416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.743616 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.743444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.743616 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.743535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844192 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p6f\" (UniqueName: \"kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.844969 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.844953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.845032 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.845006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.846823 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.846795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.853656 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.853634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p6f\" (UniqueName: \"kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:54.957144 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:54.957047    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:55.229501 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:55.229418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:42:55.232485 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:42:55.232451    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5eb3a2d_5b9f_45a9_a12d_037594548fb8.slice/crio-25f665c1858a5ea85527b3dd370ad4acf1c8bb91aa96ee2fc8e03ec96ac30fb3 WatchSource:0}: Error finding container 25f665c1858a5ea85527b3dd370ad4acf1c8bb91aa96ee2fc8e03ec96ac30fb3: Status 404 returned error can't find the container with id 25f665c1858a5ea85527b3dd370ad4acf1c8bb91aa96ee2fc8e03ec96ac30fb3
Sep 17 22:42:55.613240 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:55.613199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerStarted","Data":"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"}
Sep 17 22:42:55.613447 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:55.613268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerStarted","Data":"25f665c1858a5ea85527b3dd370ad4acf1c8bb91aa96ee2fc8e03ec96ac30fb3"}
Sep 17 22:42:56.624540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:56.624497    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerID="df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3" exitCode=0
Sep 17 22:42:56.625042 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:56.624585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerDied","Data":"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"}
Sep 17 22:42:57.630532 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.630494    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerStarted","Data":"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"}
Sep 17 22:42:57.630931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.630578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:57.630931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.630611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:42:57.630931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.630635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerStarted","Data":"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"}
Sep 17 22:42:57.632157 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.632123    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:42:57.649917 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:57.649860    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podStartSLOduration=2.564841108 podStartE2EDuration="3.649842074s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.085000966s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:42:57.648888169 +0000 UTC m=+2682.316950949" watchObservedRunningTime="2026-09-17 22:42:57.649842074 +0000 UTC m=+2682.317904821"
Sep 17 22:42:58.635474 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:58.635426    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:42:59.573059 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:42:59.572978    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:43:03.641158 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:03.641128    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:43:03.641729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:03.641692    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:43:09.572685 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:09.572636    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:43:13.642615 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:13.642571    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:43:19.572399 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:19.572356    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:43:23.642255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:23.642208    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:43:29.572687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:29.572631    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:43:33.641718 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:33.641671    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:43:39.573661 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:39.573632    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:43:43.642572 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:43.642512    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:43:47.364766 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:47.364729    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:43:47.365203 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:47.365021    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" containerID="cri-o://b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d" gracePeriod=30
Sep 17 22:43:47.365203 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:47.365042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kube-rbac-proxy" containerID="cri-o://feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001" gracePeriod=30
Sep 17 22:43:47.821392 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:47.821355    2582 generic.go:356] "Generic (PLEG): container finished" podID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerID="feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001" exitCode=2
Sep 17 22:43:47.821589 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:47.821401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerDied","Data":"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"}
Sep 17 22:43:49.568297 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:49.568246    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 22:43:49.572732 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:49.572702    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 22:43:50.595282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.595256    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:43:50.676664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.676632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") pod \"c20959a8-c144-4679-93fd-bb1ea0904e88\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") "
Sep 17 22:43:50.676850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.676685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert\") pod \"c20959a8-c144-4679-93fd-bb1ea0904e88\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") "
Sep 17 22:43:50.676850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.676716    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location\") pod \"c20959a8-c144-4679-93fd-bb1ea0904e88\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") "
Sep 17 22:43:50.676850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.676759    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rns\" (UniqueName: \"kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns\") pod \"c20959a8-c144-4679-93fd-bb1ea0904e88\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") "
Sep 17 22:43:50.676850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.676794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config\") pod \"c20959a8-c144-4679-93fd-bb1ea0904e88\" (UID: \"c20959a8-c144-4679-93fd-bb1ea0904e88\") "
Sep 17 22:43:50.677167 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.677141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c20959a8-c144-4679-93fd-bb1ea0904e88" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:43:50.677251 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.677218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c20959a8-c144-4679-93fd-bb1ea0904e88" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:43:50.678952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.678924    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c20959a8-c144-4679-93fd-bb1ea0904e88" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:43:50.679020 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.678997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns" (OuterVolumeSpecName: "kube-api-access-g2rns") pod "c20959a8-c144-4679-93fd-bb1ea0904e88" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88"). InnerVolumeSpecName "kube-api-access-g2rns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:43:50.686970 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.686913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c20959a8-c144-4679-93fd-bb1ea0904e88" (UID: "c20959a8-c144-4679-93fd-bb1ea0904e88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:50.777648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.777606    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c20959a8-c144-4679-93fd-bb1ea0904e88-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:43:50.777648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.777643    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rns\" (UniqueName: \"kubernetes.io/projected/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-api-access-g2rns\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:43:50.777648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.777657    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:43:50.777946 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.777672    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c20959a8-c144-4679-93fd-bb1ea0904e88-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:43:50.777946 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.777686    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c20959a8-c144-4679-93fd-bb1ea0904e88-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:43:50.835026 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.834990    2582 generic.go:356] "Generic (PLEG): container finished" podID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerID="b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d" exitCode=0
Sep 17 22:43:50.835026 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.835029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerDied","Data":"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"}
Sep 17 22:43:50.835260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.835053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"
Sep 17 22:43:50.835260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.835105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9" event={"ID":"c20959a8-c144-4679-93fd-bb1ea0904e88","Type":"ContainerDied","Data":"cb36c52a866f47776b2038fba57daec2ad3282e779db6f909df535340f01a8c9"}
Sep 17 22:43:50.835260 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.835133    2582 scope.go:117] "RemoveContainer" containerID="feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"
Sep 17 22:43:50.843528 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.843508    2582 scope.go:117] "RemoveContainer" containerID="b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"
Sep 17 22:43:50.851388 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.851359    2582 scope.go:117] "RemoveContainer" containerID="ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"
Sep 17 22:43:50.857155 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.857130    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:43:50.858920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.858901    2582 scope.go:117] "RemoveContainer" containerID="feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"
Sep 17 22:43:50.859249 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:43:50.859221    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001\": container with ID starting with feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001 not found: ID does not exist" containerID="feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"
Sep 17 22:43:50.859364 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.859258    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001"} err="failed to get container status \"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001\": rpc error: code = NotFound desc = could not find container \"feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001\": container with ID starting with feed5a6878284c1249c24d4ccbbd3cf06c7960568005da7d375752bb3270d001 not found: ID does not exist"
Sep 17 22:43:50.859364 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.859277    2582 scope.go:117] "RemoveContainer" containerID="b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"
Sep 17 22:43:50.859614 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:43:50.859572    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d\": container with ID starting with b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d not found: ID does not exist" containerID="b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"
Sep 17 22:43:50.859614 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.859601    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d"} err="failed to get container status \"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d\": rpc error: code = NotFound desc = could not find container \"b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d\": container with ID starting with b8d04cdde568ba26da823949aea58087103ed39430e29f57dd4424fe25eeb13d not found: ID does not exist"
Sep 17 22:43:50.859732 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.859624    2582 scope.go:117] "RemoveContainer" containerID="ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"
Sep 17 22:43:50.859928 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:43:50.859908    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a\": container with ID starting with ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a not found: ID does not exist" containerID="ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"
Sep 17 22:43:50.859997 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.859932    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a"} err="failed to get container status \"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a\": rpc error: code = NotFound desc = could not find container \"ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a\": container with ID starting with ad3e1d0ea6232c5ab717806ac4a04eaafc497d5cc512706cf22ff03014d5493a not found: ID does not exist"
Sep 17 22:43:50.861540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:50.861520    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-n5pk9"]
Sep 17 22:43:51.472696 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:51.472659    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" path="/var/lib/kubelet/pods/c20959a8-c144-4679-93fd-bb1ea0904e88/volumes"
Sep 17 22:43:53.641809 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:43:53.641773    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:44:03.641891 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:03.641845    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 22:44:13.642717 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:13.642686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:44:14.653068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:14.653034    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:44:14.653502 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:14.653286    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container" containerID="cri-o://d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065" gracePeriod=30
Sep 17 22:44:14.653502 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:14.653332    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kube-rbac-proxy" containerID="cri-o://ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39" gracePeriod=30
Sep 17 22:44:14.926890 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:14.926799    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerID="ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39" exitCode=2
Sep 17 22:44:14.926890 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:14.926872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerDied","Data":"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"}
Sep 17 22:44:15.738489 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738446    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:15.738922 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738890    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kube-rbac-proxy"
Sep 17 22:44:15.738922 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738909    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kube-rbac-proxy"
Sep 17 22:44:15.739010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738927    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="storage-initializer"
Sep 17 22:44:15.739010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738939    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="storage-initializer"
Sep 17 22:44:15.739010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738954    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container"
Sep 17 22:44:15.739010 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.738964    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container"
Sep 17 22:44:15.739183 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.739034    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kube-rbac-proxy"
Sep 17 22:44:15.739183 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.739047    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20959a8-c144-4679-93fd-bb1ea0904e88" containerName="kserve-container"
Sep 17 22:44:15.741370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.741347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.744307 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.744288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 22:44:15.744403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.744327    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 22:44:15.752159 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.752138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:15.857175 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.857139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.857370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.857187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.857370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.857246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28vr\" (UniqueName: \"kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.857370 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.857264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.957952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.957913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28vr\" (UniqueName: \"kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.957952 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.957955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.958230 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.957988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.958230 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.958019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.958230 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:15.958171    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 22:44:15.958410 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:15.958258    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls podName:bf1dd4b4-d1d6-4f08-8a23-e6778831779d nodeName:}" failed. No retries permitted until 2026-09-17 22:44:16.458236012 +0000 UTC m=+2761.126298741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" (UID: "bf1dd4b4-d1d6-4f08-8a23-e6778831779d") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 22:44:15.958410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.958387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.958689 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.958670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:15.968850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:15.968827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28vr\" (UniqueName: \"kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:16.460913 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:16.460877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:16.463398 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:16.463371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:16.652068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:16.652027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:16.924748 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:16.924716    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:16.927134 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:44:16.927106    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1dd4b4_d1d6_4f08_8a23_e6778831779d.slice/crio-d4e73e260942bb17e2b1c3b7a1bc5d66e3d0fe9037bfe1976e0742af199559d4 WatchSource:0}: Error finding container d4e73e260942bb17e2b1c3b7a1bc5d66e3d0fe9037bfe1976e0742af199559d4: Status 404 returned error can't find the container with id d4e73e260942bb17e2b1c3b7a1bc5d66e3d0fe9037bfe1976e0742af199559d4
Sep 17 22:44:16.935593 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:16.935542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerStarted","Data":"d4e73e260942bb17e2b1c3b7a1bc5d66e3d0fe9037bfe1976e0742af199559d4"}
Sep 17 22:44:17.941248 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:17.941203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerStarted","Data":"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"}
Sep 17 22:44:18.635942 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:18.635898    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 22:44:19.697124 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.697097    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:44:19.788096 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls\") pod \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") "
Sep 17 22:44:19.788300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5p6f\" (UniqueName: \"kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f\") pod \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") "
Sep 17 22:44:19.788300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788143    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config\") pod \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") "
Sep 17 22:44:19.788300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert\") pod \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") "
Sep 17 22:44:19.788300 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location\") pod \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\" (UID: \"c5eb3a2d-5b9f-45a9-a12d-037594548fb8\") "
Sep 17 22:44:19.788609 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5eb3a2d-5b9f-45a9-a12d-037594548fb8" (UID: "c5eb3a2d-5b9f-45a9-a12d-037594548fb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:44:19.788688 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5eb3a2d-5b9f-45a9-a12d-037594548fb8" (UID: "c5eb3a2d-5b9f-45a9-a12d-037594548fb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:44:19.788688 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.788652    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5eb3a2d-5b9f-45a9-a12d-037594548fb8" (UID: "c5eb3a2d-5b9f-45a9-a12d-037594548fb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:19.790273 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.790253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f" (OuterVolumeSpecName: "kube-api-access-j5p6f") pod "c5eb3a2d-5b9f-45a9-a12d-037594548fb8" (UID: "c5eb3a2d-5b9f-45a9-a12d-037594548fb8"). InnerVolumeSpecName "kube-api-access-j5p6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:44:19.790371 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.790350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5eb3a2d-5b9f-45a9-a12d-037594548fb8" (UID: "c5eb3a2d-5b9f-45a9-a12d-037594548fb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:44:19.889774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.889738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5p6f\" (UniqueName: \"kubernetes.io/projected/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-api-access-j5p6f\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:19.889774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.889768    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:19.889774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.889778    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:19.890015 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.889788    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:19.890015 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.889797    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5eb3a2d-5b9f-45a9-a12d-037594548fb8-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:19.951231 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.951195    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerID="d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065" exitCode=0
Sep 17 22:44:19.951426 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.951227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerDied","Data":"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"}
Sep 17 22:44:19.951426 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.951266    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"
Sep 17 22:44:19.951426 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.951290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp" event={"ID":"c5eb3a2d-5b9f-45a9-a12d-037594548fb8","Type":"ContainerDied","Data":"25f665c1858a5ea85527b3dd370ad4acf1c8bb91aa96ee2fc8e03ec96ac30fb3"}
Sep 17 22:44:19.951426 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.951316    2582 scope.go:117] "RemoveContainer" containerID="ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"
Sep 17 22:44:19.959377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.959361    2582 scope.go:117] "RemoveContainer" containerID="d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"
Sep 17 22:44:19.966709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.966693    2582 scope.go:117] "RemoveContainer" containerID="df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"
Sep 17 22:44:19.972896 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.972873    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:44:19.973910 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.973895    2582 scope.go:117] "RemoveContainer" containerID="ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"
Sep 17 22:44:19.974232 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:19.974185    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39\": container with ID starting with ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39 not found: ID does not exist" containerID="ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"
Sep 17 22:44:19.974367 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.974225    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39"} err="failed to get container status \"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39\": rpc error: code = NotFound desc = could not find container \"ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39\": container with ID starting with ae7d2633c3ae780e65a17017e1074c30203c533c662361608ffa1c69af5efe39 not found: ID does not exist"
Sep 17 22:44:19.974367 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.974252    2582 scope.go:117] "RemoveContainer" containerID="d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"
Sep 17 22:44:19.974660 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:19.974629    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065\": container with ID starting with d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065 not found: ID does not exist" containerID="d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"
Sep 17 22:44:19.974750 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.974670    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065"} err="failed to get container status \"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065\": rpc error: code = NotFound desc = could not find container \"d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065\": container with ID starting with d192c27a39178cfb9124ca66116250122ec812f95278675dbcdf5a5468a4f065 not found: ID does not exist"
Sep 17 22:44:19.974750 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.974695    2582 scope.go:117] "RemoveContainer" containerID="df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"
Sep 17 22:44:19.975120 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:19.974988    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3\": container with ID starting with df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3 not found: ID does not exist" containerID="df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"
Sep 17 22:44:19.975120 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.975020    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3"} err="failed to get container status \"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3\": rpc error: code = NotFound desc = could not find container \"df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3\": container with ID starting with df7b04d1ef52ac81f037738a93046e755b6e792b7dd5e23d04b1ecab125839d3 not found: ID does not exist"
Sep 17 22:44:19.976026 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:19.976005    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-568744c544-5hdxp"]
Sep 17 22:44:21.472566 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:21.472529    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" path="/var/lib/kubelet/pods/c5eb3a2d-5b9f-45a9-a12d-037594548fb8/volumes"
Sep 17 22:44:22.964874 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:22.964847    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/0.log"
Sep 17 22:44:22.965241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:22.964885    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerID="7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006" exitCode=1
Sep 17 22:44:22.965241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:22.964916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerDied","Data":"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"}
Sep 17 22:44:23.970050 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:23.970018    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/0.log"
Sep 17 22:44:23.970449 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:23.970094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerStarted","Data":"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"}
Sep 17 22:44:25.718271 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:25.718238    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:25.718760 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:25.718510    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer" containerID="cri-o://33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9" gracePeriod=30
Sep 17 22:44:26.851689 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851656    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851914    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="storage-initializer"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851924    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="storage-initializer"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851932    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851937    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851950    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kube-rbac-proxy"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.851955    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kube-rbac-proxy"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.852011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kube-rbac-proxy"
Sep 17 22:44:26.852062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.852019    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5eb3a2d-5b9f-45a9-a12d-037594548fb8" containerName="kserve-container"
Sep 17 22:44:26.853959 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.853937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:26.856683 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.856649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 22:44:26.856849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.856691    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 22:44:26.856849 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.856733    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:44:26.864823 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.864792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:44:26.946166 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.946127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:26.946166 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.946166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9jj5\" (UniqueName: \"kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:26.946390 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.946193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:26.946390 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.946289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:26.946390 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:26.946329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.046959 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.046917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.046969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9jj5\" (UniqueName: \"kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047163 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047453 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047655 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.047716 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.047639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.049518 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.049497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.055104 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.055078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9jj5\" (UniqueName: \"kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.164417 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.164313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:27.343063 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.343040    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/1.log"
Sep 17 22:44:27.343372 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.343359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/0.log"
Sep 17 22:44:27.344497 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.344484    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:27.434811 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.434720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:44:27.437447 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:44:27.437422    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ee4ea2_9cec_4073_890c_2d14f0916658.slice/crio-66c7c0801f801600999ea913269b2b4a467ff52360316f5ead2f2da10dbf7bcc WatchSource:0}: Error finding container 66c7c0801f801600999ea913269b2b4a467ff52360316f5ead2f2da10dbf7bcc: Status 404 returned error can't find the container with id 66c7c0801f801600999ea913269b2b4a467ff52360316f5ead2f2da10dbf7bcc
Sep 17 22:44:27.450414 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") pod \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") "
Sep 17 22:44:27.450485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450427    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config\") pod \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") "
Sep 17 22:44:27.450485 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location\") pod \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") "
Sep 17 22:44:27.450624 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450521    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k28vr\" (UniqueName: \"kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr\") pod \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\" (UID: \"bf1dd4b4-d1d6-4f08-8a23-e6778831779d\") "
Sep 17 22:44:27.450813 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450780    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf1dd4b4-d1d6-4f08-8a23-e6778831779d" (UID: "bf1dd4b4-d1d6-4f08-8a23-e6778831779d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:27.450929 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.450848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf1dd4b4-d1d6-4f08-8a23-e6778831779d" (UID: "bf1dd4b4-d1d6-4f08-8a23-e6778831779d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:44:27.452678 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.452653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr" (OuterVolumeSpecName: "kube-api-access-k28vr") pod "bf1dd4b4-d1d6-4f08-8a23-e6778831779d" (UID: "bf1dd4b4-d1d6-4f08-8a23-e6778831779d"). InnerVolumeSpecName "kube-api-access-k28vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:44:27.452775 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.452688    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf1dd4b4-d1d6-4f08-8a23-e6778831779d" (UID: "bf1dd4b4-d1d6-4f08-8a23-e6778831779d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:44:27.551860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.551826    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28vr\" (UniqueName: \"kubernetes.io/projected/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-api-access-k28vr\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:27.551860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.551854    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:27.551860 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.551864    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:27.552186 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.551874    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf1dd4b4-d1d6-4f08-8a23-e6778831779d-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:44:27.991134 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.991096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerStarted","Data":"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"}
Sep 17 22:44:27.991651 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.991170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerStarted","Data":"66c7c0801f801600999ea913269b2b4a467ff52360316f5ead2f2da10dbf7bcc"}
Sep 17 22:44:27.992427 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992408    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/1.log"
Sep 17 22:44:27.992792 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992777    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx_bf1dd4b4-d1d6-4f08-8a23-e6778831779d/storage-initializer/0.log"
Sep 17 22:44:27.992881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992808    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerID="33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9" exitCode=1
Sep 17 22:44:27.992881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerDied","Data":"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"}
Sep 17 22:44:27.992881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992855    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"
Sep 17 22:44:27.993054 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx" event={"ID":"bf1dd4b4-d1d6-4f08-8a23-e6778831779d","Type":"ContainerDied","Data":"d4e73e260942bb17e2b1c3b7a1bc5d66e3d0fe9037bfe1976e0742af199559d4"}
Sep 17 22:44:27.993054 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:27.992917    2582 scope.go:117] "RemoveContainer" containerID="33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"
Sep 17 22:44:28.001628 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.001609    2582 scope.go:117] "RemoveContainer" containerID="7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"
Sep 17 22:44:28.010356 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.010332    2582 scope.go:117] "RemoveContainer" containerID="33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"
Sep 17 22:44:28.010681 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:28.010658    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9\": container with ID starting with 33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9 not found: ID does not exist" containerID="33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"
Sep 17 22:44:28.010776 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.010689    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9"} err="failed to get container status \"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9\": rpc error: code = NotFound desc = could not find container \"33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9\": container with ID starting with 33d1b536b8fcdeaca7533aa97face3ed9d78c19683a2183c960ae758024708c9 not found: ID does not exist"
Sep 17 22:44:28.010776 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.010706    2582 scope.go:117] "RemoveContainer" containerID="7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"
Sep 17 22:44:28.010967 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:44:28.010948    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006\": container with ID starting with 7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006 not found: ID does not exist" containerID="7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"
Sep 17 22:44:28.011038 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.010972    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006"} err="failed to get container status \"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006\": rpc error: code = NotFound desc = could not find container \"7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006\": container with ID starting with 7c865f554d332d16d8f0c3940500cffadc05cfeb338adbd4caf1d1ad8e158006 not found: ID does not exist"
Sep 17 22:44:28.043869 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.043832    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:28.047594 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.047563    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-679ffd84cb-qqvfx"]
Sep 17 22:44:28.998850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.998752    2582 generic.go:356] "Generic (PLEG): container finished" podID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerID="b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04" exitCode=0
Sep 17 22:44:28.998850 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:28.998798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerDied","Data":"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"}
Sep 17 22:44:29.472587 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:29.472531    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" path="/var/lib/kubelet/pods/bf1dd4b4-d1d6-4f08-8a23-e6778831779d/volumes"
Sep 17 22:44:30.005213 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.005179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerStarted","Data":"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"}
Sep 17 22:44:30.005694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.005245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerStarted","Data":"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"}
Sep 17 22:44:30.005694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.005272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:30.005694 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.005327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:30.006875 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.006847    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:44:30.023902 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:30.023849    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podStartSLOduration=2.948834921 podStartE2EDuration="4.023832496s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.074997575s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:44:30.021929567 +0000 UTC m=+2774.689992314" watchObservedRunningTime="2026-09-17 22:44:30.023832496 +0000 UTC m=+2774.691895243"
Sep 17 22:44:31.009494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:31.009443    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:44:36.014083 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:36.014054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:44:36.014583 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:36.014558    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:44:46.014818 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:46.014777    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:44:56.015095 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:44:56.015055    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:06.014518 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:06.014477    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:16.014856 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:16.014815    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:26.014534 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:26.014495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:36.014929 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:36.014882    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:39.468837 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:39.468797    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:45:49.471854 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:49.471823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:45:56.874811 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:56.874775    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:45:56.875363 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:56.875069    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" containerID="cri-o://a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067" gracePeriod=30
Sep 17 22:45:56.875363 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:56.875181    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kube-rbac-proxy" containerID="cri-o://e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be" gracePeriod=30
Sep 17 22:45:57.325158 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:57.325125    2582 generic.go:356] "Generic (PLEG): container finished" podID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerID="e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be" exitCode=2
Sep 17 22:45:57.325376 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:57.325205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerDied","Data":"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"}
Sep 17 22:45:58.011615 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.011574    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.011980    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.012000    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.012015    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.012023    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.012105    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.012132 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.012116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1dd4b4-d1d6-4f08-8a23-e6778831779d" containerName="storage-initializer"
Sep 17 22:45:58.015127 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.015105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.018463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.018434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 22:45:58.018617 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.018497    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 22:45:58.028017 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.027992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:45:58.131151 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.131121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.131335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.131159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.131335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.131192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.131335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.131223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pdc9\" (UniqueName: \"kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.232295 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.232245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.232295 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.232292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.232608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.232336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.232608 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.232370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pdc9\" (UniqueName: \"kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.232608 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:45:58.232539    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 22:45:58.232773 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:45:58.232625    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls podName:c01a0c2c-27c1-44c7-ba2f-abd6ee581091 nodeName:}" failed. No retries permitted until 2026-09-17 22:45:58.732603065 +0000 UTC m=+2863.400665809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" (UID: "c01a0c2c-27c1-44c7-ba2f-abd6ee581091") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 22:45:58.232837 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.232786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.233039 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.233017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.241200 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.241173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pdc9\" (UniqueName: \"kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.737540 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.737496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.740013 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.739986    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:58.925684 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:58.925641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:45:59.199471 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:59.199392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:45:59.202895 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:45:59.202869    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01a0c2c_27c1_44c7_ba2f_abd6ee581091.slice/crio-53bdc0420e262a289cde9e2b100b56d813b1a32230077212b6f4d230d3b9b5df WatchSource:0}: Error finding container 53bdc0420e262a289cde9e2b100b56d813b1a32230077212b6f4d230d3b9b5df: Status 404 returned error can't find the container with id 53bdc0420e262a289cde9e2b100b56d813b1a32230077212b6f4d230d3b9b5df
Sep 17 22:45:59.333506 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:59.333460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerStarted","Data":"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"}
Sep 17 22:45:59.333709 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:59.333572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerStarted","Data":"53bdc0420e262a289cde9e2b100b56d813b1a32230077212b6f4d230d3b9b5df"}
Sep 17 22:45:59.469212 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:45:59.469120    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 22:46:01.010133 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:01.010089    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 22:46:02.031596 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.031541    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:46:02.063745 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.063706    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config\") pod \"88ee4ea2-9cec-4073-890c-2d14f0916658\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") "
Sep 17 22:46:02.063934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.063760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9jj5\" (UniqueName: \"kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5\") pod \"88ee4ea2-9cec-4073-890c-2d14f0916658\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") "
Sep 17 22:46:02.063934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.063814    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location\") pod \"88ee4ea2-9cec-4073-890c-2d14f0916658\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") "
Sep 17 22:46:02.063934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.063841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert\") pod \"88ee4ea2-9cec-4073-890c-2d14f0916658\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") "
Sep 17 22:46:02.063934 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.063868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls\") pod \"88ee4ea2-9cec-4073-890c-2d14f0916658\" (UID: \"88ee4ea2-9cec-4073-890c-2d14f0916658\") "
Sep 17 22:46:02.064172 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.064147    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88ee4ea2-9cec-4073-890c-2d14f0916658" (UID: "88ee4ea2-9cec-4073-890c-2d14f0916658"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:46:02.064231 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.064186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88ee4ea2-9cec-4073-890c-2d14f0916658" (UID: "88ee4ea2-9cec-4073-890c-2d14f0916658"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:46:02.064231 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.064195    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88ee4ea2-9cec-4073-890c-2d14f0916658" (UID: "88ee4ea2-9cec-4073-890c-2d14f0916658"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:46:02.066037 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.066003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5" (OuterVolumeSpecName: "kube-api-access-j9jj5") pod "88ee4ea2-9cec-4073-890c-2d14f0916658" (UID: "88ee4ea2-9cec-4073-890c-2d14f0916658"). InnerVolumeSpecName "kube-api-access-j9jj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:46:02.066149 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.066045    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88ee4ea2-9cec-4073-890c-2d14f0916658" (UID: "88ee4ea2-9cec-4073-890c-2d14f0916658"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:46:02.164988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.164893    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88ee4ea2-9cec-4073-890c-2d14f0916658-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:02.164988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.164931    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:02.164988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.164943    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ea2-9cec-4073-890c-2d14f0916658-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:02.164988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.164951    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:02.164988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.164959    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9jj5\" (UniqueName: \"kubernetes.io/projected/88ee4ea2-9cec-4073-890c-2d14f0916658-kube-api-access-j9jj5\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:02.346760 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.346724    2582 generic.go:356] "Generic (PLEG): container finished" podID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerID="a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067" exitCode=0
Sep 17 22:46:02.346935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.346780    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"
Sep 17 22:46:02.346935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.346803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerDied","Data":"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"}
Sep 17 22:46:02.346935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.346862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft" event={"ID":"88ee4ea2-9cec-4073-890c-2d14f0916658","Type":"ContainerDied","Data":"66c7c0801f801600999ea913269b2b4a467ff52360316f5ead2f2da10dbf7bcc"}
Sep 17 22:46:02.346935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.346889    2582 scope.go:117] "RemoveContainer" containerID="e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"
Sep 17 22:46:02.355973 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.355951    2582 scope.go:117] "RemoveContainer" containerID="a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"
Sep 17 22:46:02.364052 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.364029    2582 scope.go:117] "RemoveContainer" containerID="b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"
Sep 17 22:46:02.369967 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.369940    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:46:02.372810 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.372786    2582 scope.go:117] "RemoveContainer" containerID="e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"
Sep 17 22:46:02.373140 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:46:02.373122    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be\": container with ID starting with e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be not found: ID does not exist" containerID="e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"
Sep 17 22:46:02.373208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373148    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be"} err="failed to get container status \"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be\": rpc error: code = NotFound desc = could not find container \"e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be\": container with ID starting with e4e52f48d9e099c9e02144d003f9990b1cd9c10209238b0c12de7b00e74955be not found: ID does not exist"
Sep 17 22:46:02.373208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373168    2582 scope.go:117] "RemoveContainer" containerID="a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"
Sep 17 22:46:02.373309 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373258    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-766c9cdf8d-zv2ft"]
Sep 17 22:46:02.373440 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:46:02.373413    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067\": container with ID starting with a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067 not found: ID does not exist" containerID="a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"
Sep 17 22:46:02.373509 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373451    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067"} err="failed to get container status \"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067\": rpc error: code = NotFound desc = could not find container \"a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067\": container with ID starting with a4173d90cec617a43453c013a9d4742fffc42336818efb50c249e0dad825e067 not found: ID does not exist"
Sep 17 22:46:02.373509 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373480    2582 scope.go:117] "RemoveContainer" containerID="b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"
Sep 17 22:46:02.373812 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:46:02.373793    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04\": container with ID starting with b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04 not found: ID does not exist" containerID="b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"
Sep 17 22:46:02.373869 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:02.373818    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04"} err="failed to get container status \"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04\": rpc error: code = NotFound desc = could not find container \"b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04\": container with ID starting with b5a85ebb4e7fbd3b6b15fb765fb4b32e357b0669a734137dcf9a1c8f880f0b04 not found: ID does not exist"
Sep 17 22:46:03.472584 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:03.472531    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" path="/var/lib/kubelet/pods/88ee4ea2-9cec-4073-890c-2d14f0916658/volumes"
Sep 17 22:46:05.360237 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:05.360209    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/0.log"
Sep 17 22:46:05.360663 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:05.360247    2582 generic.go:356] "Generic (PLEG): container finished" podID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerID="56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998" exitCode=1
Sep 17 22:46:05.360663 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:05.360302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerDied","Data":"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"}
Sep 17 22:46:06.365666 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:06.365632    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/0.log"
Sep 17 22:46:06.366212 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:06.365693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerStarted","Data":"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"}
Sep 17 22:46:07.973429 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:07.973396    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:46:07.973863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:07.973659    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer" containerID="cri-o://bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea" gracePeriod=30
Sep 17 22:46:09.092375 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092337    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092752    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092769    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092784    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="storage-initializer"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092795    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="storage-initializer"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092804    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kube-rbac-proxy"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092812    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kube-rbac-proxy"
Sep 17 22:46:09.092876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092876    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kube-rbac-proxy"
Sep 17 22:46:09.093243 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.092887    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ee4ea2-9cec-4073-890c-2d14f0916658" containerName="kserve-container"
Sep 17 22:46:09.096073 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.096053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.098774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.098748    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:46:09.098893 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.098768    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 22:46:09.098893 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.098802    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 22:46:09.108016 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.107990    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:46:09.218038 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.218000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.218222 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.218047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.218222 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.218078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpt97\" (UniqueName: \"kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.218327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.218204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.218327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.218249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319397 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319728 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319728 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319728 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpt97\" (UniqueName: \"kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.319983 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.319963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.320158 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.320141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.320256 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.320235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.322040 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.322021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.328038 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.328017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpt97\" (UniqueName: \"kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.406681 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.406584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:09.679500 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:09.679473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:46:09.682000 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:46:09.681971    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7fa89d_afe9_4a00_9075_12cc6fa3bd30.slice/crio-66e4166969767ccffe35178914bbe12dab33ce33f8e87c81aa71e0232f053325 WatchSource:0}: Error finding container 66e4166969767ccffe35178914bbe12dab33ce33f8e87c81aa71e0232f053325: Status 404 returned error can't find the container with id 66e4166969767ccffe35178914bbe12dab33ce33f8e87c81aa71e0232f053325
Sep 17 22:46:10.380918 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:10.380886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerStarted","Data":"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"}
Sep 17 22:46:10.381292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:10.380943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerStarted","Data":"66e4166969767ccffe35178914bbe12dab33ce33f8e87c81aa71e0232f053325"}
Sep 17 22:46:11.386480 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:11.386439    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerID="b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f" exitCode=0
Sep 17 22:46:11.386928 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:11.386530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerDied","Data":"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"}
Sep 17 22:46:12.010932 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.010906    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/1.log"
Sep 17 22:46:12.011333 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.011317    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/0.log"
Sep 17 22:46:12.013586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.013570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:46:12.144449 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config\") pod \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") "
Sep 17 22:46:12.144635 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144463    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") pod \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") "
Sep 17 22:46:12.144635 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location\") pod \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") "
Sep 17 22:46:12.144635 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pdc9\" (UniqueName: \"kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9\") pod \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\" (UID: \"c01a0c2c-27c1-44c7-ba2f-abd6ee581091\") "
Sep 17 22:46:12.144843 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c01a0c2c-27c1-44c7-ba2f-abd6ee581091" (UID: "c01a0c2c-27c1-44c7-ba2f-abd6ee581091"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:46:12.144881 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.144829    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c01a0c2c-27c1-44c7-ba2f-abd6ee581091" (UID: "c01a0c2c-27c1-44c7-ba2f-abd6ee581091"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:46:12.146662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.146635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9" (OuterVolumeSpecName: "kube-api-access-8pdc9") pod "c01a0c2c-27c1-44c7-ba2f-abd6ee581091" (UID: "c01a0c2c-27c1-44c7-ba2f-abd6ee581091"). InnerVolumeSpecName "kube-api-access-8pdc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:46:12.146662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.146652    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c01a0c2c-27c1-44c7-ba2f-abd6ee581091" (UID: "c01a0c2c-27c1-44c7-ba2f-abd6ee581091"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:46:12.245079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.244990    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:12.245079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.245019    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:12.245079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.245029    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:12.245079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.245038    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pdc9\" (UniqueName: \"kubernetes.io/projected/c01a0c2c-27c1-44c7-ba2f-abd6ee581091-kube-api-access-8pdc9\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:46:12.392111 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392084    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/1.log"
Sep 17 22:46:12.392492 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j_c01a0c2c-27c1-44c7-ba2f-abd6ee581091/storage-initializer/0.log"
Sep 17 22:46:12.392543 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392507    2582 generic.go:356] "Generic (PLEG): container finished" podID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerID="bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea" exitCode=1
Sep 17 22:46:12.392621 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392586    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"
Sep 17 22:46:12.392677 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerDied","Data":"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"}
Sep 17 22:46:12.392736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j" event={"ID":"c01a0c2c-27c1-44c7-ba2f-abd6ee581091","Type":"ContainerDied","Data":"53bdc0420e262a289cde9e2b100b56d813b1a32230077212b6f4d230d3b9b5df"}
Sep 17 22:46:12.392798 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.392737    2582 scope.go:117] "RemoveContainer" containerID="bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"
Sep 17 22:46:12.395091 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.395066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerStarted","Data":"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"}
Sep 17 22:46:12.395238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.395130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerStarted","Data":"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"}
Sep 17 22:46:12.395238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.395170    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:12.395238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.395215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:12.396523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.396497    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:12.401197 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.401178    2582 scope.go:117] "RemoveContainer" containerID="56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"
Sep 17 22:46:12.409415 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.409397    2582 scope.go:117] "RemoveContainer" containerID="bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"
Sep 17 22:46:12.409716 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:46:12.409699    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea\": container with ID starting with bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea not found: ID does not exist" containerID="bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"
Sep 17 22:46:12.409774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.409724    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea"} err="failed to get container status \"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea\": rpc error: code = NotFound desc = could not find container \"bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea\": container with ID starting with bd51792c55ba8aeb602b35d90340d33bdfdb59e2fa8c0967ff699065eac54cea not found: ID does not exist"
Sep 17 22:46:12.409774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.409740    2582 scope.go:117] "RemoveContainer" containerID="56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"
Sep 17 22:46:12.410013 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:46:12.409995    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998\": container with ID starting with 56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998 not found: ID does not exist" containerID="56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"
Sep 17 22:46:12.410068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.410015    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998"} err="failed to get container status \"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998\": rpc error: code = NotFound desc = could not find container \"56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998\": container with ID starting with 56fcc2f05c059dd4134462b239af711dc41fd91ffb87b2ba8fa3b36916c4d998 not found: ID does not exist"
Sep 17 22:46:12.420897 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.420850    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podStartSLOduration=2.335486882 podStartE2EDuration="3.420834411s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.085347529s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:46:12.419634237 +0000 UTC m=+2877.087697002" watchObservedRunningTime="2026-09-17 22:46:12.420834411 +0000 UTC m=+2877.088897158"
Sep 17 22:46:12.449343 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.449311    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:46:12.453772 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:12.453742    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-65d6648d9b-l8m7j"]
Sep 17 22:46:13.399615 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:13.399564    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:13.472239 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:13.472205    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" path="/var/lib/kubelet/pods/c01a0c2c-27c1-44c7-ba2f-abd6ee581091/volumes"
Sep 17 22:46:18.409964 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:18.409933    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:46:18.410410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:18.410384    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:28.411126 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:28.411087    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:38.410487 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:38.410447    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:48.410945 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:48.410896    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:46:58.410624 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:46:58.410581    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:47:08.410743 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:08.410696    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:47:18.410800 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:18.410757    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 22:47:28.411238 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:28.411204    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:47:29.093599 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:29.093531    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:47:29.094072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:29.093826    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container" containerID="cri-o://b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d" gracePeriod=30
Sep 17 22:47:29.094072 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:29.093840    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kube-rbac-proxy" containerID="cri-o://5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673" gracePeriod=30
Sep 17 22:47:29.674801 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:29.674764    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerID="5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673" exitCode=2
Sep 17 22:47:29.674801 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:29.674805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerDied","Data":"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"}
Sep 17 22:47:30.188023 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.187986    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:30.188258 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188247    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.188304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188260    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.188304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188273    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.188304 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188280    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.188403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188322    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.188403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.188330    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01a0c2c-27c1-44c7-ba2f-abd6ee581091" containerName="storage-initializer"
Sep 17 22:47:30.191270 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.191253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.194195 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.194169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 22:47:30.194306 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.194219    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 22:47:30.202864 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.202841    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:30.215831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.215802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhbq\" (UniqueName: \"kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.215969 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.215852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.215969 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.215919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.215969 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.215949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.316311 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.316266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhbq\" (UniqueName: \"kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.316311 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.316323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.316586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.316350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.316586 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.316368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.316786 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.316768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.317078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.317050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.318909 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.318891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.325388 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.325364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhbq\" (UniqueName: \"kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.500877 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.500785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:30.776359 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:30.776325    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:30.778919 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:47:30.778893    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb553202_3d32_4edc_b35d_c72dbf91f8a3.slice/crio-700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b WatchSource:0}: Error finding container 700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b: Status 404 returned error can't find the container with id 700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b
Sep 17 22:47:31.683777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:31.683739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerStarted","Data":"14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"}
Sep 17 22:47:31.683974 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:31.683796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerStarted","Data":"700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b"}
Sep 17 22:47:33.400106 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:33.400067    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 22:47:34.135060 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.135026    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:47:34.247724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.247631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location\") pod \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") "
Sep 17 22:47:34.247724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.247672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config\") pod \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") "
Sep 17 22:47:34.247724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.247689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert\") pod \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") "
Sep 17 22:47:34.247724 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.247724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls\") pod \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") "
Sep 17 22:47:34.247988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.247752    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpt97\" (UniqueName: \"kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97\") pod \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\" (UID: \"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30\") "
Sep 17 22:47:34.248066 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.248047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" (UID: "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:47:34.248111 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.248064    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" (UID: "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:47:34.248111 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.248039    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" (UID: "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:47:34.249866 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.249840    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97" (OuterVolumeSpecName: "kube-api-access-kpt97") pod "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" (UID: "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30"). InnerVolumeSpecName "kube-api-access-kpt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:47:34.249945 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.249925    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" (UID: "7e7fa89d-afe9-4a00-9075-12cc6fa3bd30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:47:34.349244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.349201    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:34.349244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.349237    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:34.349244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.349246    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:34.349244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.349255    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:34.349513 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.349263    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpt97\" (UniqueName: \"kubernetes.io/projected/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30-kube-api-access-kpt97\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:34.697486 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.697447    2582 generic.go:356] "Generic (PLEG): container finished" podID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerID="b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d" exitCode=0
Sep 17 22:47:34.698078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.697502    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"
Sep 17 22:47:34.698078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.697527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerDied","Data":"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"}
Sep 17 22:47:34.698078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.697610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m" event={"ID":"7e7fa89d-afe9-4a00-9075-12cc6fa3bd30","Type":"ContainerDied","Data":"66e4166969767ccffe35178914bbe12dab33ce33f8e87c81aa71e0232f053325"}
Sep 17 22:47:34.698078 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.697637    2582 scope.go:117] "RemoveContainer" containerID="5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"
Sep 17 22:47:34.707251 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.707232    2582 scope.go:117] "RemoveContainer" containerID="b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"
Sep 17 22:47:34.715265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.715243    2582 scope.go:117] "RemoveContainer" containerID="b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"
Sep 17 22:47:34.721116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.721088    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:47:34.724177 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724157    2582 scope.go:117] "RemoveContainer" containerID="5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"
Sep 17 22:47:34.724570 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:34.724524    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673\": container with ID starting with 5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673 not found: ID does not exist" containerID="5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"
Sep 17 22:47:34.724679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724580    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673"} err="failed to get container status \"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673\": rpc error: code = NotFound desc = could not find container \"5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673\": container with ID starting with 5becc43ec386af41c8016ac45163b75c936f520ff9d17844fa04d892323c7673 not found: ID does not exist"
Sep 17 22:47:34.724679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724606    2582 scope.go:117] "RemoveContainer" containerID="b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"
Sep 17 22:47:34.724679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724636    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-87p6m"]
Sep 17 22:47:34.724927 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:34.724909    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d\": container with ID starting with b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d not found: ID does not exist" containerID="b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"
Sep 17 22:47:34.724995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724937    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d"} err="failed to get container status \"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d\": rpc error: code = NotFound desc = could not find container \"b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d\": container with ID starting with b6006b0d7ad89d945c158c339db71f5c843fefbe2d7c0b8fb14044f4b294916d not found: ID does not exist"
Sep 17 22:47:34.724995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.724962    2582 scope.go:117] "RemoveContainer" containerID="b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"
Sep 17 22:47:34.725224 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:34.725208    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f\": container with ID starting with b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f not found: ID does not exist" containerID="b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"
Sep 17 22:47:34.725274 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:34.725228    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f"} err="failed to get container status \"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f\": rpc error: code = NotFound desc = could not find container \"b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f\": container with ID starting with b05b0caeb583fa76b9d4c8862285c63a9579266886f46e3e0f5ad17206cd891f not found: ID does not exist"
Sep 17 22:47:35.472336 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:35.472300    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" path="/var/lib/kubelet/pods/7e7fa89d-afe9-4a00-9075-12cc6fa3bd30/volumes"
Sep 17 22:47:35.703094 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:35.703068    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/0.log"
Sep 17 22:47:35.703475 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:35.703104    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerID="14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6" exitCode=1
Sep 17 22:47:35.703475 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:35.703130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerDied","Data":"14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"}
Sep 17 22:47:36.708315 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:36.708288    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/0.log"
Sep 17 22:47:36.708749 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:36.708342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerStarted","Data":"505e61ad4c773de166d3c23e41dac7bff43cd9aa8c94209aa2734cbc1ebd2a7d"}
Sep 17 22:47:38.718181 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.718155    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/1.log"
Sep 17 22:47:38.718702 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.718613    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/0.log"
Sep 17 22:47:38.718702 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.718644    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerID="505e61ad4c773de166d3c23e41dac7bff43cd9aa8c94209aa2734cbc1ebd2a7d" exitCode=1
Sep 17 22:47:38.718834 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.718717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerDied","Data":"505e61ad4c773de166d3c23e41dac7bff43cd9aa8c94209aa2734cbc1ebd2a7d"}
Sep 17 22:47:38.718834 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.718782    2582 scope.go:117] "RemoveContainer" containerID="14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"
Sep 17 22:47:38.719142 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.719107    2582 scope.go:117] "RemoveContainer" containerID="14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"
Sep 17 22:47:38.729236 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:38.729205    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_kserve-ci-e2e-test(bb553202-3d32-4edc-b35d-c72dbf91f8a3)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3"
Sep 17 22:47:38.729375 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:38.729347    2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_kserve-ci-e2e-test_bb553202-3d32-4edc-b35d-c72dbf91f8a3_0 in pod sandbox 700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b from index: no such id: '14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6'" containerID="14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"
Sep 17 22:47:38.729461 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:38.729386    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_kserve-ci-e2e-test_bb553202-3d32-4edc-b35d-c72dbf91f8a3_0 in pod sandbox 700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b from index: no such id: '14717c5c7dc8ee49b626b2bb6056287f60a50056b91ac75af25dd9c1b3c391d6'"
Sep 17 22:47:39.724270 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:39.724240    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/1.log"
Sep 17 22:47:40.167742 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.167703    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:40.297387 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.297362    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/1.log"
Sep 17 22:47:40.300840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.300824    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:40.395463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395426    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbhbq\" (UniqueName: \"kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq\") pod \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") "
Sep 17 22:47:40.395669 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls\") pod \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") "
Sep 17 22:47:40.395669 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config\") pod \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") "
Sep 17 22:47:40.395669 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location\") pod \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\" (UID: \"bb553202-3d32-4edc-b35d-c72dbf91f8a3\") "
Sep 17 22:47:40.395935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395906    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb553202-3d32-4edc-b35d-c72dbf91f8a3" (UID: "bb553202-3d32-4edc-b35d-c72dbf91f8a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:47:40.396014 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.395913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb553202-3d32-4edc-b35d-c72dbf91f8a3" (UID: "bb553202-3d32-4edc-b35d-c72dbf91f8a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:47:40.397732 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.397708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb553202-3d32-4edc-b35d-c72dbf91f8a3" (UID: "bb553202-3d32-4edc-b35d-c72dbf91f8a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:47:40.397826 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.397709    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq" (OuterVolumeSpecName: "kube-api-access-kbhbq") pod "bb553202-3d32-4edc-b35d-c72dbf91f8a3" (UID: "bb553202-3d32-4edc-b35d-c72dbf91f8a3"). InnerVolumeSpecName "kube-api-access-kbhbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:47:40.496762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.496652    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb553202-3d32-4edc-b35d-c72dbf91f8a3-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:40.496762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.496691    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:40.496762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.496706    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:40.496762 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.496717    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhbq\" (UniqueName: \"kubernetes.io/projected/bb553202-3d32-4edc-b35d-c72dbf91f8a3-kube-api-access-kbhbq\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:47:40.729785 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.729761    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7_bb553202-3d32-4edc-b35d-c72dbf91f8a3/storage-initializer/1.log"
Sep 17 22:47:40.730215 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.729808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7" event={"ID":"bb553202-3d32-4edc-b35d-c72dbf91f8a3","Type":"ContainerDied","Data":"700770d71b80f3fe38f6e38f52d027a6c58f6e1c378b7742ca3e3fb06db0845b"}
Sep 17 22:47:40.730215 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.729861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"
Sep 17 22:47:40.730215 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.729872    2582 scope.go:117] "RemoveContainer" containerID="505e61ad4c773de166d3c23e41dac7bff43cd9aa8c94209aa2734cbc1ebd2a7d"
Sep 17 22:47:40.765354 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.765278    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:40.769607 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:40.769539    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7479f45c57-pvgk7"]
Sep 17 22:47:41.338410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338376    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:47:41.338648 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338636    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338649    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338666    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338671    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338680    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338687    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="storage-initializer"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338698    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338707    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kube-rbac-proxy"
Sep 17 22:47:41.338712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338712    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kube-rbac-proxy"
Sep 17 22:47:41.338991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338753    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338761    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" containerName="storage-initializer"
Sep 17 22:47:41.338991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338767    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kube-rbac-proxy"
Sep 17 22:47:41.338991 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.338773    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7fa89d-afe9-4a00-9075-12cc6fa3bd30" containerName="kserve-container"
Sep 17 22:47:41.343018 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.343001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.346065 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.346043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:47:41.346198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.346131    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 22:47:41.346198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.346143    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:47:41.346198 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.346160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:47:41.346347 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.346253    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:47:41.347255 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.347234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:47:41.347341 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.347234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 22:47:41.353044 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.353020    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:47:41.403241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.403198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfs5\" (UniqueName: \"kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.403241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.403240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.403470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.403367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.403470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.403438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.403579 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.403475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.472377 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.472345    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb553202-3d32-4edc-b35d-c72dbf91f8a3" path="/var/lib/kubelet/pods/bb553202-3d32-4edc-b35d-c72dbf91f8a3/volumes"
Sep 17 22:47:41.504321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.504466 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.504466 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfs5\" (UniqueName: \"kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.504621 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:41.504463    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 22:47:41.504621 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.504621 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:47:41.504528    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls podName:c3443b09-cdbd-42c3-b44f-51a1acb6c58b nodeName:}" failed. No retries permitted until 2026-09-17 22:47:42.004512448 +0000 UTC m=+2966.672575174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 22:47:41.504774 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.504900 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.504881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.505130 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.505113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.505199 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.505183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:41.513364 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:41.513334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfs5\" (UniqueName: \"kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:42.008515 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.008453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:42.010937 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.010916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:42.254716 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.254675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:42.531920 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.531861    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:47:42.534056 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:47:42.534029    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3443b09_cdbd_42c3_b44f_51a1acb6c58b.slice/crio-2ceae702891eca2ee30478c96c56e26b3e8151de14a511e5dfbe9af08ebe8d46 WatchSource:0}: Error finding container 2ceae702891eca2ee30478c96c56e26b3e8151de14a511e5dfbe9af08ebe8d46: Status 404 returned error can't find the container with id 2ceae702891eca2ee30478c96c56e26b3e8151de14a511e5dfbe9af08ebe8d46
Sep 17 22:47:42.740995 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.740959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerStarted","Data":"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"}
Sep 17 22:47:42.741148 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:42.741021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerStarted","Data":"2ceae702891eca2ee30478c96c56e26b3e8151de14a511e5dfbe9af08ebe8d46"}
Sep 17 22:47:46.756604 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:46.756568    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerID="4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f" exitCode=0
Sep 17 22:47:46.756996 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:46.756636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerDied","Data":"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"}
Sep 17 22:47:46.757899 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:46.757883    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:47:54.795339 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:54.795300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerStarted","Data":"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"}
Sep 17 22:47:54.795783 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:54.795366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerStarted","Data":"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"}
Sep 17 22:47:54.795783 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:54.795673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:54.815275 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:54.815219    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podStartSLOduration=2.565047689 podStartE2EDuration="13.815202578s" totalImagesPullingTime="7.184340969s" totalInitContainerRuntime="4.06581392s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:47:54.813777822 +0000 UTC m=+2979.481840619" watchObservedRunningTime="2026-09-17 22:47:54.815202578 +0000 UTC m=+2979.483265325"
Sep 17 22:47:55.799527 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:55.799491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:47:55.800784 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:55.800753    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:47:56.802980 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:47:56.802942    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:01.808427 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:01.808396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:48:01.808990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:01.808965    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:11.809999 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:11.809959    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:12.782090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.782043    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:48:12.789528 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.789504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:12.792365 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.792343    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 22:48:12.792465 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.792380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:48:12.795310 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.795283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:48:12.956470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.956428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:12.956470 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.956475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:12.956979 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.956503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:12.956979 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.956605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:12.956979 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:12.956657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfsq\" (UniqueName: \"kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.057798 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.057766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.057988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.057805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.057988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.057827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfsq\" (UniqueName: \"kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.057988 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.057863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.058160 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.058128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.058233 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.058215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.058662 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.058640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.058714 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.058640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.060723 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.060700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.066712 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.066685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfsq\" (UniqueName: \"kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq\") pod \"isvc-pmml-runtime-predictor-885fc8777-hn4m6\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.100691 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.100649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:13.377530 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.377450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:48:13.380480 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:48:13.380452    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84089452_a0e8_42c4_9619_b6ed8a4102f3.slice/crio-b565ab22be04864d2f84fde74fa6761c05fa5bb3a4c83ea9ac5ce535f740fa7c WatchSource:0}: Error finding container b565ab22be04864d2f84fde74fa6761c05fa5bb3a4c83ea9ac5ce535f740fa7c: Status 404 returned error can't find the container with id b565ab22be04864d2f84fde74fa6761c05fa5bb3a4c83ea9ac5ce535f740fa7c
Sep 17 22:48:13.868690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.868651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerStarted","Data":"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"}
Sep 17 22:48:13.868877 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:13.868713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerStarted","Data":"b565ab22be04864d2f84fde74fa6761c05fa5bb3a4c83ea9ac5ce535f740fa7c"}
Sep 17 22:48:17.885722 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:17.885687    2582 generic.go:356] "Generic (PLEG): container finished" podID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerID="5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28" exitCode=0
Sep 17 22:48:17.886116 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:17.885739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerDied","Data":"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"}
Sep 17 22:48:18.891969 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:18.891934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerStarted","Data":"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"}
Sep 17 22:48:18.892409 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:18.891994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerStarted","Data":"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"}
Sep 17 22:48:18.892409 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:18.892246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:18.912633 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:18.912585    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podStartSLOduration=2.854533196 podStartE2EDuration="6.912569919s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.058036723s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:48:18.911096648 +0000 UTC m=+3003.579159394" watchObservedRunningTime="2026-09-17 22:48:18.912569919 +0000 UTC m=+3003.580632713"
Sep 17 22:48:19.896808 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:19.896775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:19.898235 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:19.898209    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:48:20.901081 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:20.901042    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:48:21.808992 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:21.808952    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:25.906143 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:25.906113    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:48:25.906639 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:25.906607    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:48:31.809062 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:31.809022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:35.907245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:35.907198    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:48:41.809892 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:41.809847    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:45.907112 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:45.907070    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:48:51.809368 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:51.809327    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:48:55.907259 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:48:55.907221    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:01.809265 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:01.809174    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:49:05.907282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:05.907238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:08.468342 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:08.468299    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:49:15.907311 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:15.907271    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:18.468324 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:18.468275    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 22:49:25.907415 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:25.907377    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:28.468729 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:28.468698    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:49:33.146568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:33.146511    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:49:33.146974 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:33.146823    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container" containerID="cri-o://618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b" gracePeriod=30
Sep 17 22:49:33.146974 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:33.146855    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kube-rbac-proxy" containerID="cri-o://1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5" gracePeriod=30
Sep 17 22:49:34.180418 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:34.180386    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerID="1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5" exitCode=2
Sep 17 22:49:34.180819 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:34.180453    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerDied","Data":"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"}
Sep 17 22:49:35.907110 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:35.907064    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:36.803642 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:36.803588    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 17 22:49:37.989944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:37.989916    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:49:38.020483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") pod \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") "
Sep 17 22:49:38.020690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location\") pod \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") "
Sep 17 22:49:38.020690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert\") pod \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") "
Sep 17 22:49:38.020690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config\") pod \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") "
Sep 17 22:49:38.020690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020615    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbfs5\" (UniqueName: \"kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5\") pod \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\" (UID: \"c3443b09-cdbd-42c3-b44f-51a1acb6c58b\") "
Sep 17 22:49:38.020942 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3443b09-cdbd-42c3-b44f-51a1acb6c58b" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:49:38.020942 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3443b09-cdbd-42c3-b44f-51a1acb6c58b" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:49:38.020942 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.020917    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3443b09-cdbd-42c3-b44f-51a1acb6c58b" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:49:38.022716 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.022686    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3443b09-cdbd-42c3-b44f-51a1acb6c58b" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:49:38.022831 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.022772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5" (OuterVolumeSpecName: "kube-api-access-cbfs5") pod "c3443b09-cdbd-42c3-b44f-51a1acb6c58b" (UID: "c3443b09-cdbd-42c3-b44f-51a1acb6c58b"). InnerVolumeSpecName "kube-api-access-cbfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:49:38.121690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.121604    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:49:38.121690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.121631    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:49:38.121690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.121643    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:49:38.121690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.121652    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:49:38.121690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.121660    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbfs5\" (UniqueName: \"kubernetes.io/projected/c3443b09-cdbd-42c3-b44f-51a1acb6c58b-kube-api-access-cbfs5\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:49:38.196298 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.196261    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerID="618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b" exitCode=0
Sep 17 22:49:38.196483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.196311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerDied","Data":"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"}
Sep 17 22:49:38.196483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.196314    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"
Sep 17 22:49:38.196483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.196358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks" event={"ID":"c3443b09-cdbd-42c3-b44f-51a1acb6c58b","Type":"ContainerDied","Data":"2ceae702891eca2ee30478c96c56e26b3e8151de14a511e5dfbe9af08ebe8d46"}
Sep 17 22:49:38.196483 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.196385    2582 scope.go:117] "RemoveContainer" containerID="1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"
Sep 17 22:49:38.204847 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.204829    2582 scope.go:117] "RemoveContainer" containerID="618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"
Sep 17 22:49:38.212124 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.212107    2582 scope.go:117] "RemoveContainer" containerID="4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"
Sep 17 22:49:38.219523 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.219494    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:49:38.220252 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.220236    2582 scope.go:117] "RemoveContainer" containerID="1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"
Sep 17 22:49:38.220720 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:49:38.220682    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5\": container with ID starting with 1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5 not found: ID does not exist" containerID="1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"
Sep 17 22:49:38.220815 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.220730    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5"} err="failed to get container status \"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5\": rpc error: code = NotFound desc = could not find container \"1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5\": container with ID starting with 1c73602ad33e7ec15f3faef11af3d2bf47809b2d134e5b0e1a7990bd4d343ce5 not found: ID does not exist"
Sep 17 22:49:38.220815 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.220750    2582 scope.go:117] "RemoveContainer" containerID="618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"
Sep 17 22:49:38.221123 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:49:38.221097    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b\": container with ID starting with 618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b not found: ID does not exist" containerID="618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"
Sep 17 22:49:38.221220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.221130    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b"} err="failed to get container status \"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b\": rpc error: code = NotFound desc = could not find container \"618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b\": container with ID starting with 618bd5be7194a638313af63e351b143333b7cfc91031f6739883d7c6c74d220b not found: ID does not exist"
Sep 17 22:49:38.221220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.221152    2582 scope.go:117] "RemoveContainer" containerID="4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"
Sep 17 22:49:38.221480 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:49:38.221461    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f\": container with ID starting with 4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f not found: ID does not exist" containerID="4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"
Sep 17 22:49:38.221541 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.221483    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f"} err="failed to get container status \"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f\": rpc error: code = NotFound desc = could not find container \"4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f\": container with ID starting with 4c64825945a6e32221216b8d5f9cfe76b728d0873f249a8b918bbbe8151c061f not found: ID does not exist"
Sep 17 22:49:38.222674 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.222655    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-l2jks"]
Sep 17 22:49:38.468568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:38.468446    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:39.473043 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:39.472998    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" path="/var/lib/kubelet/pods/c3443b09-cdbd-42c3-b44f-51a1acb6c58b/volumes"
Sep 17 22:49:48.468627 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:48.468582    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:49:58.468785 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:49:58.468730    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:50:04.744966 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:04.744930    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:50:04.745468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:04.745170    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" containerID="cri-o://a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa" gracePeriod=30
Sep 17 22:50:04.745468 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:04.745230    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kube-rbac-proxy" containerID="cri-o://4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3" gracePeriod=30
Sep 17 22:50:05.296011 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:05.295972    2582 generic.go:356] "Generic (PLEG): container finished" podID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerID="4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3" exitCode=2
Sep 17 22:50:05.296196 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:05.296045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerDied","Data":"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"}
Sep 17 22:50:05.901756 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:05.901708    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 17 22:50:08.468467 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:08.468415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 22:50:09.294292 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.294266    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:50:09.311935 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.311904    2582 generic.go:356] "Generic (PLEG): container finished" podID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerID="a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa" exitCode=0
Sep 17 22:50:09.312088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.311957    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"
Sep 17 22:50:09.312088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.311983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerDied","Data":"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"}
Sep 17 22:50:09.312088 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.312067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6" event={"ID":"84089452-a0e8-42c4-9619-b6ed8a4102f3","Type":"ContainerDied","Data":"b565ab22be04864d2f84fde74fa6761c05fa5bb3a4c83ea9ac5ce535f740fa7c"}
Sep 17 22:50:09.312220 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.312094    2582 scope.go:117] "RemoveContainer" containerID="4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"
Sep 17 22:50:09.322328 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.322304    2582 scope.go:117] "RemoveContainer" containerID="a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"
Sep 17 22:50:09.330092 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.330065    2582 scope.go:117] "RemoveContainer" containerID="5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"
Sep 17 22:50:09.337159 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.337136    2582 scope.go:117] "RemoveContainer" containerID="4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"
Sep 17 22:50:09.337444 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:50:09.337424    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3\": container with ID starting with 4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3 not found: ID does not exist" containerID="4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"
Sep 17 22:50:09.337495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.337455    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3"} err="failed to get container status \"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3\": rpc error: code = NotFound desc = could not find container \"4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3\": container with ID starting with 4d6e6197e8e8e943861db43f586abd957c085029515014ba0493437f792cbcc3 not found: ID does not exist"
Sep 17 22:50:09.337495 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.337472    2582 scope.go:117] "RemoveContainer" containerID="a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"
Sep 17 22:50:09.337753 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:50:09.337736    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa\": container with ID starting with a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa not found: ID does not exist" containerID="a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"
Sep 17 22:50:09.337800 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.337761    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa"} err="failed to get container status \"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa\": rpc error: code = NotFound desc = could not find container \"a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa\": container with ID starting with a4aad58b0354b3e27efbe657f348b451eab2210c62df4bb906f91a472a5311aa not found: ID does not exist"
Sep 17 22:50:09.337800 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.337780    2582 scope.go:117] "RemoveContainer" containerID="5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"
Sep 17 22:50:09.338043 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:50:09.338023    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28\": container with ID starting with 5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28 not found: ID does not exist" containerID="5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"
Sep 17 22:50:09.338090 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.338054    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28"} err="failed to get container status \"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28\": rpc error: code = NotFound desc = could not find container \"5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28\": container with ID starting with 5f42e9208256f47c54d73647090abb0a486f127af11314a44f20c0f51458fd28 not found: ID does not exist"
Sep 17 22:50:09.352393 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config\") pod \"84089452-a0e8-42c4-9619-b6ed8a4102f3\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") "
Sep 17 22:50:09.352505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsfsq\" (UniqueName: \"kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq\") pod \"84089452-a0e8-42c4-9619-b6ed8a4102f3\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") "
Sep 17 22:50:09.352505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert\") pod \"84089452-a0e8-42c4-9619-b6ed8a4102f3\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") "
Sep 17 22:50:09.352505 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352477    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls\") pod \"84089452-a0e8-42c4-9619-b6ed8a4102f3\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") "
Sep 17 22:50:09.352650 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352516    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location\") pod \"84089452-a0e8-42c4-9619-b6ed8a4102f3\" (UID: \"84089452-a0e8-42c4-9619-b6ed8a4102f3\") "
Sep 17 22:50:09.352773 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84089452-a0e8-42c4-9619-b6ed8a4102f3" (UID: "84089452-a0e8-42c4-9619-b6ed8a4102f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:50:09.352904 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84089452-a0e8-42c4-9619-b6ed8a4102f3" (UID: "84089452-a0e8-42c4-9619-b6ed8a4102f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:50:09.352971 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.352916    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84089452-a0e8-42c4-9619-b6ed8a4102f3" (UID: "84089452-a0e8-42c4-9619-b6ed8a4102f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:09.354686 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.354653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84089452-a0e8-42c4-9619-b6ed8a4102f3" (UID: "84089452-a0e8-42c4-9619-b6ed8a4102f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:50:09.354785 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.354716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq" (OuterVolumeSpecName: "kube-api-access-hsfsq") pod "84089452-a0e8-42c4-9619-b6ed8a4102f3" (UID: "84089452-a0e8-42c4-9619-b6ed8a4102f3"). InnerVolumeSpecName "kube-api-access-hsfsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:50:09.453318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.453221    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfsq\" (UniqueName: \"kubernetes.io/projected/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-api-access-hsfsq\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:50:09.453318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.453263    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:50:09.453318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.453272    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84089452-a0e8-42c4-9619-b6ed8a4102f3-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:50:09.453318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.453304    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84089452-a0e8-42c4-9619-b6ed8a4102f3-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:50:09.453318 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.453313    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84089452-a0e8-42c4-9619-b6ed8a4102f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:50:09.630463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.630426    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:50:09.636673 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.636644    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hn4m6"]
Sep 17 22:50:09.894244 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894206    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:50:09.894458 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894446    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894459    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894474    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894480    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894489    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="storage-initializer"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="storage-initializer"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894501    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kube-rbac-proxy"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894506    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kube-rbac-proxy"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894515    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kube-rbac-proxy"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894521    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kube-rbac-proxy"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894528    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="storage-initializer"
Sep 17 22:50:09.894568 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894535    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="storage-initializer"
Sep 17 22:50:09.894901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894604    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kserve-container"
Sep 17 22:50:09.894901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894615    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kserve-container"
Sep 17 22:50:09.894901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894621    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3443b09-cdbd-42c3-b44f-51a1acb6c58b" containerName="kube-rbac-proxy"
Sep 17 22:50:09.894901 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.894627    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" containerName="kube-rbac-proxy"
Sep 17 22:50:09.898970 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.898951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:09.901944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.901864    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-347c27-predictor-serving-cert\""
Sep 17 22:50:09.901944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.901883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-347c27-kube-rbac-proxy-sar-config\""
Sep 17 22:50:09.901944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.901902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:50:09.901944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.901922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:50:09.902282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.902210    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:50:09.902282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.902253    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:50:09.902394 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.902294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:50:09.907321 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.907289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:50:09.957414 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.957378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:09.957414 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.957417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:09.957664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.957481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:09.957664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.957540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8b75\" (UniqueName: \"kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:09.957664 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:09.957610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.058707 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.058666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8b75\" (UniqueName: \"kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.058840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.058714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.058840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.058745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.058840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.058767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.058840 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.058794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.059173 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.059153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.059449 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.059427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.059572 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.059530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.061299 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.061282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.067687 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.067663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8b75\" (UniqueName: \"kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75\") pod \"isvc-primary-347c27-predictor-6868659584-g4fnk\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") " pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.209326 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.209231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:10.483036 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:10.482949    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:50:10.486141 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:50:10.486111    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5160927a_357f_4e96_8610_43f8383475a6.slice/crio-ad518d472596cdf538c9951ca96dac6cf1cf7ce2a32aef6d0582c1cca5f06a11 WatchSource:0}: Error finding container ad518d472596cdf538c9951ca96dac6cf1cf7ce2a32aef6d0582c1cca5f06a11: Status 404 returned error can't find the container with id ad518d472596cdf538c9951ca96dac6cf1cf7ce2a32aef6d0582c1cca5f06a11
Sep 17 22:50:11.320463 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:11.320429    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerStarted","Data":"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"}
Sep 17 22:50:11.320867 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:11.320486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerStarted","Data":"ad518d472596cdf538c9951ca96dac6cf1cf7ce2a32aef6d0582c1cca5f06a11"}
Sep 17 22:50:11.472843 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:11.472809    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84089452-a0e8-42c4-9619-b6ed8a4102f3" path="/var/lib/kubelet/pods/84089452-a0e8-42c4-9619-b6ed8a4102f3/volumes"
Sep 17 22:50:15.335949 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:15.335916    2582 generic.go:356] "Generic (PLEG): container finished" podID="5160927a-357f-4e96-8610-43f8383475a6" containerID="8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9" exitCode=0
Sep 17 22:50:15.336400 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:15.336000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerDied","Data":"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"}
Sep 17 22:50:16.342241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:16.342209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerStarted","Data":"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"}
Sep 17 22:50:16.342654 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:16.342272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerStarted","Data":"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"}
Sep 17 22:50:16.342654 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:16.342471    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:16.362583 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:16.362520    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podStartSLOduration=3.582603735 podStartE2EDuration="7.362505984s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.779902249s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:50:16.360531143 +0000 UTC m=+3121.028593890" watchObservedRunningTime="2026-09-17 22:50:16.362505984 +0000 UTC m=+3121.030568731"
Sep 17 22:50:17.345629 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:17.345585    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:17.346626 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:17.346599    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:50:18.348679 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:18.348642    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:50:23.354267 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:23.354238    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:50:23.354931 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:23.354901    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:50:33.355673 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:33.355588    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:50:43.355033 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:43.354994    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:50:53.355808 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:50:53.355769    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:51:03.355506 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:51:03.355458    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:51:13.355313 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:51:13.355265    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:51:23.355241 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:51:23.355204    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:51:27.470164 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:51:27.470121    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 22:51:37.472537 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:51:37.472510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:52:11.155871 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:11.155785    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:52:11.156450 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:11.156099    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container" containerID="cri-o://f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1" gracePeriod=30
Sep 17 22:52:11.156450 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:11.156164    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kube-rbac-proxy" containerID="cri-o://cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266" gracePeriod=30
Sep 17 22:52:11.764297 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:11.764258    2582 generic.go:356] "Generic (PLEG): container finished" podID="5160927a-357f-4e96-8610-43f8383475a6" containerID="cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266" exitCode=2
Sep 17 22:52:11.764473 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:11.764328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerDied","Data":"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"}
Sep 17 22:52:13.349858 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:13.349813    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 22:52:15.998739 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:15.998711    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:52:16.032034 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config\") pod \"5160927a-357f-4e96-8610-43f8383475a6\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") "
Sep 17 22:52:16.032034 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert\") pod \"5160927a-357f-4e96-8610-43f8383475a6\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") "
Sep 17 22:52:16.032245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8b75\" (UniqueName: \"kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75\") pod \"5160927a-357f-4e96-8610-43f8383475a6\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") "
Sep 17 22:52:16.032245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032083    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location\") pod \"5160927a-357f-4e96-8610-43f8383475a6\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") "
Sep 17 22:52:16.032245 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls\") pod \"5160927a-357f-4e96-8610-43f8383475a6\" (UID: \"5160927a-357f-4e96-8610-43f8383475a6\") "
Sep 17 22:52:16.032423 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5160927a-357f-4e96-8610-43f8383475a6" (UID: "5160927a-357f-4e96-8610-43f8383475a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:52:16.032469 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5160927a-357f-4e96-8610-43f8383475a6" (UID: "5160927a-357f-4e96-8610-43f8383475a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:52:16.032469 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.032441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5160927a-357f-4e96-8610-43f8383475a6" (UID: "5160927a-357f-4e96-8610-43f8383475a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:52:16.034214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.034190    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75" (OuterVolumeSpecName: "kube-api-access-r8b75") pod "5160927a-357f-4e96-8610-43f8383475a6" (UID: "5160927a-357f-4e96-8610-43f8383475a6"). InnerVolumeSpecName "kube-api-access-r8b75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:52:16.034283 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.034260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5160927a-357f-4e96-8610-43f8383475a6" (UID: "5160927a-357f-4e96-8610-43f8383475a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:52:16.132857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.132745    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:52:16.132857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.132794    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5160927a-357f-4e96-8610-43f8383475a6-cabundle-cert\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:52:16.132857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.132809    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8b75\" (UniqueName: \"kubernetes.io/projected/5160927a-357f-4e96-8610-43f8383475a6-kube-api-access-r8b75\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:52:16.132857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.132821    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5160927a-357f-4e96-8610-43f8383475a6-kserve-provision-location\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:52:16.132857 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.132834    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5160927a-357f-4e96-8610-43f8383475a6-proxy-tls\") on node \"ip-10-0-136-9.ec2.internal\" DevicePath \"\""
Sep 17 22:52:16.783649 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.783608    2582 generic.go:356] "Generic (PLEG): container finished" podID="5160927a-357f-4e96-8610-43f8383475a6" containerID="f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1" exitCode=0
Sep 17 22:52:16.783862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.783659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerDied","Data":"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"}
Sep 17 22:52:16.783862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.783717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk" event={"ID":"5160927a-357f-4e96-8610-43f8383475a6","Type":"ContainerDied","Data":"ad518d472596cdf538c9951ca96dac6cf1cf7ce2a32aef6d0582c1cca5f06a11"}
Sep 17 22:52:16.783862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.783665    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"
Sep 17 22:52:16.783862 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.783750    2582 scope.go:117] "RemoveContainer" containerID="cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"
Sep 17 22:52:16.792048 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.792029    2582 scope.go:117] "RemoveContainer" containerID="f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"
Sep 17 22:52:16.801045 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.801024    2582 scope.go:117] "RemoveContainer" containerID="8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"
Sep 17 22:52:16.805214 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.805190    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:52:16.808876 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.808858    2582 scope.go:117] "RemoveContainer" containerID="cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"
Sep 17 22:52:16.809066 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809046    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-347c27-predictor-6868659584-g4fnk"]
Sep 17 22:52:16.809201 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:52:16.809178    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266\": container with ID starting with cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266 not found: ID does not exist" containerID="cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"
Sep 17 22:52:16.809250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809212    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266"} err="failed to get container status \"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266\": rpc error: code = NotFound desc = could not find container \"cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266\": container with ID starting with cb9f71347e97bf4b9333285e9d43f8651e0c28fb6cbdb66068b7c3e37f6de266 not found: ID does not exist"
Sep 17 22:52:16.809250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809232    2582 scope.go:117] "RemoveContainer" containerID="f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"
Sep 17 22:52:16.809559 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:52:16.809525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1\": container with ID starting with f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1 not found: ID does not exist" containerID="f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"
Sep 17 22:52:16.809715 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809687    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1"} err="failed to get container status \"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1\": rpc error: code = NotFound desc = could not find container \"f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1\": container with ID starting with f9016dc0e267a1ec94b633cbb62ab2956eb97567ccfec88a714e5153261d5ea1 not found: ID does not exist"
Sep 17 22:52:16.809715 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809714    2582 scope.go:117] "RemoveContainer" containerID="8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"
Sep 17 22:52:16.809944 ip-10-0-136-9 kubenswrapper[2582]: E0917 22:52:16.809926    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9\": container with ID starting with 8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9 not found: ID does not exist" containerID="8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"
Sep 17 22:52:16.809990 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:16.809947    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9"} err="failed to get container status \"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9\": rpc error: code = NotFound desc = could not find container \"8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9\": container with ID starting with 8515c0d124f62923a5c344e69b71f49ce2e14e8c04a3597481f644e8ffd50de9 not found: ID does not exist"
Sep 17 22:52:17.471944 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:17.471903    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5160927a-357f-4e96-8610-43f8383475a6" path="/var/lib/kubelet/pods/5160927a-357f-4e96-8610-43f8383475a6/volumes"
Sep 17 22:52:52.771067 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771028    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfvtg/must-gather-xdsvt"]
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="storage-initializer"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="storage-initializer"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771290    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kube-rbac-proxy"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771296    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kube-rbac-proxy"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771308    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771313    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771358    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kube-rbac-proxy"
Sep 17 22:52:52.771504 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.771369    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5160927a-357f-4e96-8610-43f8383475a6" containerName="kserve-container"
Sep 17 22:52:52.774159 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.774139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:52.776858 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.776834    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfvtg\"/\"openshift-service-ca.crt\""
Sep 17 22:52:52.776997 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.776977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfvtg\"/\"kube-root-ca.crt\""
Sep 17 22:52:52.777063 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.777001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfvtg\"/\"default-dockercfg-8n8bw\""
Sep 17 22:52:52.785219 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.785189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/must-gather-xdsvt"]
Sep 17 22:52:52.898164 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.898123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-must-gather-output\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:52.898350 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.898179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhbg\" (UniqueName: \"kubernetes.io/projected/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-kube-api-access-fhhbg\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:52.999524 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.999486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhbg\" (UniqueName: \"kubernetes.io/projected/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-kube-api-access-fhhbg\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:52.999736 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.999589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-must-gather-output\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:52.999903 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:52.999887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-must-gather-output\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:53.008346 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:53.008311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhbg\" (UniqueName: \"kubernetes.io/projected/4d1fd893-6a0c-4c15-9db9-ad569ed4e15f-kube-api-access-fhhbg\") pod \"must-gather-xdsvt\" (UID: \"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f\") " pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:53.083403 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:53.083366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/must-gather-xdsvt"
Sep 17 22:52:53.353208 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:53.352970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/must-gather-xdsvt"]
Sep 17 22:52:53.356021 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:52:53.355989    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d1fd893_6a0c_4c15_9db9_ad569ed4e15f.slice/crio-a4cc015ac16e9889e5557946ee06237043b9e17dec920486c5d8fce54ea378b3 WatchSource:0}: Error finding container a4cc015ac16e9889e5557946ee06237043b9e17dec920486c5d8fce54ea378b3: Status 404 returned error can't find the container with id a4cc015ac16e9889e5557946ee06237043b9e17dec920486c5d8fce54ea378b3
Sep 17 22:52:53.358029 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:53.358010    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:52:53.915288 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:53.915251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/must-gather-xdsvt" event={"ID":"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f","Type":"ContainerStarted","Data":"a4cc015ac16e9889e5557946ee06237043b9e17dec920486c5d8fce54ea378b3"}
Sep 17 22:52:54.921659 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:54.921538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/must-gather-xdsvt" event={"ID":"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f","Type":"ContainerStarted","Data":"d759bc0503b7d0cabc83106559b8d2cbe1f4e289cfd5defb1bc38f655f488372"}
Sep 17 22:52:54.921659 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:54.921627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/must-gather-xdsvt" event={"ID":"4d1fd893-6a0c-4c15-9db9-ad569ed4e15f","Type":"ContainerStarted","Data":"ea600216123f3f72db52b3250a233b3a653420c4a0c14819ac7d902e68e7ab85"}
Sep 17 22:52:55.681916 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:55.681886    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-st4hm_3397b63b-b850-4a86-9411-754f199657fe/global-pull-secret-syncer/0.log"
Sep 17 22:52:55.806019 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:55.805980    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-chhxg_568d748e-e14b-4c0c-b275-3807424561b2/konnectivity-agent/0.log"
Sep 17 22:52:55.874690 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:52:55.874660    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-9.ec2.internal_49246e01e0be5af303c7318bdf33cbba/haproxy/0.log"
Sep 17 22:53:00.437863 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:00.437826    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-jlr7j_0ea5405c-b3bf-4b09-96b8-66292f9d03f5/monitoring-plugin/0.log"
Sep 17 22:53:00.627044 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:00.627012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pk7hm_dee56a9e-9adc-44aa-bea2-5e22784fb30d/node-exporter/0.log"
Sep 17 22:53:00.650327 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:00.650302    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pk7hm_dee56a9e-9adc-44aa-bea2-5e22784fb30d/kube-rbac-proxy/0.log"
Sep 17 22:53:00.673128 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:00.673100    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pk7hm_dee56a9e-9adc-44aa-bea2-5e22784fb30d/init-textfile/0.log"
Sep 17 22:53:03.556777 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:03.556740    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c94f9f957-spw7w_038e29b7-e3eb-4676-8fbd-891d72c58ff6/console/0.log"
Sep 17 22:53:04.667440 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.667055    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfvtg/must-gather-xdsvt" podStartSLOduration=11.898259836 podStartE2EDuration="12.667027887s" totalImagesPullingTime="768.768051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:52:54.961924777 +0000 UTC m=+3279.629987525" watchObservedRunningTime="2026-09-17 22:53:04.667027887 +0000 UTC m=+3289.335090635"
Sep 17 22:53:04.668438 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.668399    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"]
Sep 17 22:53:04.674319 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.674281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.681721 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.681688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"]
Sep 17 22:53:04.800105 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.800057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-lib-modules\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.800282 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.800119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-proc\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.800335 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.800281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549d4\" (UniqueName: \"kubernetes.io/projected/8a3f6c33-fd15-455c-bded-ec92e6177246-kube-api-access-549d4\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.800387 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.800361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-podres\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.800434 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.800398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-sys\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.900790 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-podres\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.900790 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-sys\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-lib-modules\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-proc\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549d4\" (UniqueName: \"kubernetes.io/projected/8a3f6c33-fd15-455c-bded-ec92e6177246-kube-api-access-549d4\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-sys\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-podres\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.900988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-proc\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.901068 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.901012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3f6c33-fd15-455c-bded-ec92e6177246-lib-modules\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.913479 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.912238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-549d4\" (UniqueName: \"kubernetes.io/projected/8a3f6c33-fd15-455c-bded-ec92e6177246-kube-api-access-549d4\") pod \"perf-node-gather-daemonset-zkgwx\" (UID: \"8a3f6c33-fd15-455c-bded-ec92e6177246\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:04.987004 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:04.986908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:05.133250 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.133222    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rrh9f_d1351b34-0a6a-499c-95cf-f5c292809054/dns/0.log"
Sep 17 22:53:05.155293 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.155264    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rrh9f_d1351b34-0a6a-499c-95cf-f5c292809054/kube-rbac-proxy/0.log"
Sep 17 22:53:05.184307 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.184282    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7fqc_c1330de1-e458-48dd-87f7-0376f228c47f/dns-node-resolver/0.log"
Sep 17 22:53:05.285636 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.284620    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"]
Sep 17 22:53:05.290038 ip-10-0-136-9 kubenswrapper[2582]: W0917 22:53:05.289993    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a3f6c33_fd15_455c_bded_ec92e6177246.slice/crio-a5dcb80be05f0be12da8e9319a338440465d8f3621938b5d1366f0424ebbbbd9 WatchSource:0}: Error finding container a5dcb80be05f0be12da8e9319a338440465d8f3621938b5d1366f0424ebbbbd9: Status 404 returned error can't find the container with id a5dcb80be05f0be12da8e9319a338440465d8f3621938b5d1366f0424ebbbbd9
Sep 17 22:53:05.786748 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.786717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lzzd9_39743950-452b-46ba-95ce-b69a6ea9c790/node-ca/0.log"
Sep 17 22:53:05.972398 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.972346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx" event={"ID":"8a3f6c33-fd15-455c-bded-ec92e6177246","Type":"ContainerStarted","Data":"249b5d4fe4918e4616cc6a336178d96a9eec1bbc8d5e6cecf452a1b443401902"}
Sep 17 22:53:05.972612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.972419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx" event={"ID":"8a3f6c33-fd15-455c-bded-ec92e6177246","Type":"ContainerStarted","Data":"a5dcb80be05f0be12da8e9319a338440465d8f3621938b5d1366f0424ebbbbd9"}
Sep 17 22:53:05.972612 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.972539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:05.989835 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:05.989765    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx" podStartSLOduration=1.9897429770000001 podStartE2EDuration="1.989742977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:53:05.988371362 +0000 UTC m=+3290.656434157" watchObservedRunningTime="2026-09-17 22:53:05.989742977 +0000 UTC m=+3290.657805725"
Sep 17 22:53:06.880640 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:06.880605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-skjpn_a607f5d6-e98b-497c-b6c6-68f68e0bb1e6/serve-healthcheck-canary/0.log"
Sep 17 22:53:07.388363 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:07.388338    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkfpl_32d5d2b7-74a2-4e98-8344-89d03e643638/kube-rbac-proxy/0.log"
Sep 17 22:53:07.409374 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:07.409351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkfpl_32d5d2b7-74a2-4e98-8344-89d03e643638/exporter/0.log"
Sep 17 22:53:07.430289 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:07.430260    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkfpl_32d5d2b7-74a2-4e98-8344-89d03e643638/extractor/0.log"
Sep 17 22:53:09.782494 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:09.782460    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-hnklk_7b51e998-3ced-444a-9e03-a562fc35977f/manager/0.log"
Sep 17 22:53:11.988925 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:11.988873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-zkgwx"
Sep 17 22:53:15.765359 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.765332    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/kube-multus-additional-cni-plugins/0.log"
Sep 17 22:53:15.793566 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.793527    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/egress-router-binary-copy/0.log"
Sep 17 22:53:15.820079 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.820051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/cni-plugins/0.log"
Sep 17 22:53:15.843205 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.843174    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/bond-cni-plugin/0.log"
Sep 17 22:53:15.866894 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.866865    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/routeoverride-cni/0.log"
Sep 17 22:53:15.891290 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.891234    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/whereabouts-cni-bincopy/0.log"
Sep 17 22:53:15.913975 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:15.913937    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w2tk2_e1b685a8-ec3e-4a39-92f6-deb21d8dba13/whereabouts-cni/0.log"
Sep 17 22:53:16.390246 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:16.390216    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vjwsx_f7a63491-4f67-4d45-a7bc-1a5aa5a1b26f/kube-multus/0.log"
Sep 17 22:53:16.415703 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:16.415662    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-24p4h_cfbff581-3aee-4f8c-89e8-e47b010af5fd/network-metrics-daemon/0.log"
Sep 17 22:53:16.435376 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:16.435347    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-24p4h_cfbff581-3aee-4f8c-89e8-e47b010af5fd/kube-rbac-proxy/0.log"
Sep 17 22:53:17.462109 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.462071    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/ovn-controller/0.log"
Sep 17 22:53:17.479363 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.479336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/ovn-acl-logging/0.log"
Sep 17 22:53:17.502230 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.502199    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/kube-rbac-proxy-node/0.log"
Sep 17 22:53:17.522833 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.522801    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 22:53:17.546120 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.546087    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/northd/0.log"
Sep 17 22:53:17.567926 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.567892    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/nbdb/0.log"
Sep 17 22:53:17.590121 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.590089    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/sbdb/0.log"
Sep 17 22:53:17.636270 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:17.636228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5pjh7_c7fe7daa-f235-4567-afaa-f31ed3614839/ovnkube-controller/0.log"
Sep 17 22:53:18.903874 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:18.903841    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jkjz9_4819ba77-53a1-4a53-9643-699d42acdff5/network-check-target-container/0.log"
Sep 17 22:53:19.794989 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:19.794961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qx7qc_b58dc0db-afc2-4c56-9f65-161b881d7266/iptables-alerter/0.log"
Sep 17 22:53:20.505410 ip-10-0-136-9 kubenswrapper[2582]: I0917 22:53:20.505383    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-sq789_d7eaf875-1ad0-4a74-86d2-b7432d69d44d/tuned/0.log"